Big Data & Analytics

Lambda Architecture

Nathan Marz's Lambda Architecture: batch layer (HDFS + Spark), speed layer (Spark Streaming / Storm), serving layer merge pattern, dual code path complexity, and why Kappa Architecture emerged as a simpler alternative.

⏱ 10 min read

What it is

Lambda Architecture, proposed by Nathan Marz around 2011, is a data processing architecture designed to handle both high-volume historical data and real-time streaming data simultaneously. It achieves this by running two parallel processing paths — a batch layer that periodically recomputes comprehensive, accurate views over all historical data, and a speed layer that processes the incoming stream in near real-time to fill the latency gap between batch runs. A serving layer merges views from both layers to answer queries.

The architecture's core insight is that batch processing (MapReduce/Spark over HDFS) can produce arbitrarily accurate results given enough time, while real-time processing (Storm, Spark Streaming) can produce approximate results with low latency. By combining both, Lambda provides both accuracy and freshness.

Why it exists

In the early 2010s, the dominant big data processing model was batch: periodic Hadoop MapReduce jobs over HDFS. This produced accurate results but with hours of latency. Newer use cases (recommendation engines, fraud detection, operational dashboards) required data freshness in seconds or minutes, not hours. Lambda Architecture emerged as a pragmatic solution: keep the battle-tested batch pipeline for accuracy and correctness, and add a real-time layer on top for freshness, merging the results at query time.

When to use

  • Systems where both real-time approximations and accurate historical reprocessing are required.
  • Legacy contexts where a Hadoop batch pipeline already exists and the requirement is to add near real-time capability without replacing the batch system.
  • Use cases where batch reprocessing from raw data is required for corrections, regulatory compliance, or algorithm updates.

When not to use

  • New systems where the team has streaming expertise — Kappa Architecture is simpler.
  • Use cases where the data volume is small enough for a single streaming pipeline to handle reprocessing easily.
  • Teams without capacity to maintain two separate codebases implementing the same business logic.

Typical architecture


LAMBDA ARCHITECTURE
────────────────────
  All incoming data → Master dataset (append-only, immutable)
                           │
              ┌────────────┴──────────────┐
              │                           │
        Batch Layer                  Speed Layer
   (HDFS + Spark / Hive)         (Spark Streaming / Storm / Flink)
              │                           │
    Batch views (complete,          Real-time views (recent,
    accurate, hours of latency)     approximate, seconds latency)
              │                           │
              └────────────┬──────────────┘
                           │
                   Serving Layer
              (Cassandra + HBase / Druid)
                           │
                     Query results
                (batch view for history +
                 speed view for recent data)

BATCH LAYER (truth layer)
──────────────────────────
  Storage: HDFS / S3 (raw data, immutable)
  Processing: Apache Spark (formerly Hadoop MapReduce)
  Schedule: hourly or daily full recomputation
  Output: pre-computed batch views (aggregations, join results)
  Stored in: Hive tables, Parquet on S3, HBase

  Properties:
  - Always recomputes from raw data (no incremental state)
  - Accurate: re-processing corrects any errors
  - Latency: high (hours) — acceptable for historical queries

SPEED LAYER (real-time layer)
───────────────────────────────
  Input: same event stream as batch layer
  Processing: Spark Structured Streaming, Apache Storm, Flink
  Output: incremental updates covering most recent period
  Stored in: Cassandra, Redis (TTL'd), HBase (time-windowed)

  Properties:
  - Only covers data since last batch view
  - Approximate: state may contain duplicates, no corrections
  - Latency: low (seconds) — covers the "batch gap"
  - State discarded when batch view catches up

SERVING LAYER
──────────────
  Merges batch view (for data older than N hours) with
  real-time view (for data within the last N hours)
  Query: SELECT * FROM batch_view WHERE timestamp < T-N
  UNION SELECT * FROM speed_view WHERE timestamp >= T-N

DUAL CODE PATH PROBLEM
───────────────────────
  The same business logic (e.g., "user session aggregation") must be
  implemented TWICE:
    - Once in Spark (batch): run over full history
    - Once in Spark Streaming or Storm (speed): run over stream

  Two implementations → divergence:
    - Bug fixed in one, not the other
    - Different libraries, different edge case handling
    - Two test suites, two deployment pipelines

  This is the primary reason Kappa Architecture emerged as a replacement.

Pros and cons

Pros

  • Batch layer provides strong accuracy guarantees — periodic recomputation from raw data corrects any processing errors automatically.
  • Speed layer provides low latency for recent data without waiting for the next batch run.
  • Fault tolerant: if the speed layer produces wrong results, the batch layer will eventually overwrite them.
  • Well-proven pattern with mature tooling (Spark, Hive, Hadoop) and a large body of operational knowledge.

Cons

  • Dual code path: the same computation must be implemented twice (once for batch, once for stream) — divergence and bugs are common.
  • High operational complexity: two processing engines, two storage systems for views, and a serving layer merging them all.
  • Batch recomputation over large datasets can be slow and resource-intensive, delaying corrections.
  • Kappa Architecture (single stream path with replayable log) solves the dual code path problem and is simpler for most new systems.

Implementation notes

Data immutability: Lambda Architecture's correctness guarantee depends on the master dataset being immutable and append-only. Raw data must never be updated or deleted — corrections are applied by re-running the batch pipeline over the original raw data. This means the raw storage layer (HDFS / S3) must never be mutated; corrections are handled in the processing logic, not by modifying source data.

Common failure modes

  • Speed and batch view divergence: The two implementations drift apart over time as the team fixes bugs in one but not the other; build comprehensive integration tests that run both paths against the same data and verify identical results.
  • Serving layer merge complexity: Determining the exact cutoff point between batch and speed views requires careful coordination; off-by-one errors lead to either gaps or double-counting in query results.

Decision checklist

  • Is there a genuine requirement for both real-time approximations AND accurate historical reprocessing? (If only one, use a simpler architecture.)
  • Is the team prepared to maintain two separate implementations of the same business logic?
  • Has Kappa Architecture been evaluated as a simpler alternative using a replayable log (Kafka)?
  • Is the master dataset stored immutably with no in-place updates?

Example use cases

  • Social network analytics: Batch layer computes daily aggregates (total likes, follower counts) from HDFS; speed layer computes counts for the last hour; serving layer merges both for "total likes in the last 7 days" queries.
  • E-commerce recommendation engine: Batch layer trains collaborative filtering model nightly over full purchase history; speed layer updates session-level product interactions in real time; recommendations merge both signals.
  • Kappa Architecture — Simpler alternative: single streaming pipeline with replayable log eliminates the dual code path.
  • Spark Architecture — Spark is the primary engine for both the batch and speed layers in modern Lambda deployments.
  • Data Pipeline Patterns — Medallion architecture and incremental processing patterns that complement Lambda.

Further reading