Big Data & Analytics

Kappa Architecture

Jay Kreps' Kappa Architecture: stream-only processing using a replayable immutable log (Kafka), reprocessing via new consumer groups, comparison with Lambda Architecture, and when single-pipeline simplicity is the right choice.

⏱ 10 min read

What it is

Kappa Architecture, proposed by Jay Kreps (co-creator of Kafka) in a 2014 blog post titled "Questioning the Lambda Architecture", simplifies big data processing by eliminating the separate batch layer. Instead of maintaining two parallel processing pipelines (batch + streaming), Kappa uses a single streaming pipeline and relies on a durable, replayable event log (typically Apache Kafka with long retention) as the source of truth. Reprocessing — the core capability of Lambda's batch layer — is achieved in Kappa by spinning up a new stream processing job that reads from the beginning of the Kafka log.

The key insight is that a replayable log with long retention (days, weeks, or indefinitely via compaction or tiered storage) gives you the same ability to reprocess historical data as an HDFS batch layer, without requiring a separate batch processing engine and the dual code path it implies.

Why it exists

Lambda Architecture's biggest operational burden is the dual code path: the same computation must be written twice — once for the batch layer (Spark, Hadoop) and once for the speed layer (Spark Streaming, Storm). These implementations inevitably drift apart: a bug is fixed in one but not the other, libraries diverge, edge cases are handled differently. Kappa eliminates this by using a single stream processing engine for both real-time processing and historical reprocessing.

When to use

  • New systems where streaming expertise is available and the team prefers operational simplicity.
  • Use cases where business logic changes frequently — updating a single stream processor is far simpler than updating two.
  • Systems where full historical reprocessing is occasional (triggered by logic changes) rather than continuous.
  • When Kafka (or an equivalent replayable log) is already in the stack — it provides the reprocessing capability that makes Kappa viable.

When not to use

  • Workloads requiring complex multi-pass batch algorithms (ML training over full history, graph algorithms) — stream processing has limited state model compared to full Spark batch.
  • When data volume exceeds practical Kafka retention costs (though Kafka Tiered Storage and solutions like WarpStream address this).
  • When migrating an existing Lambda system — the migration cost may outweigh the simplicity benefit unless the dual code path is causing real operational problems.

Typical architecture


KAPPA ARCHITECTURE
───────────────────
  Events → Kafka (immutable, ordered, long-retention log)
                │
     ┌──────────┴──────────────────────┐
     │  Stream Processing Job (v1)     │
     │  (Flink / Spark Structured      │
     │   Streaming / Kafka Streams)    │
     └──────────┬──────────────────────┘
                │
         Serving Store
      (Cassandra, Redis, DB)
                │
           Query Results

REPROCESSING FLOW (when logic changes)
────────────────────────────────────────
  1. Deploy new Stream Processing Job v2
     - Points to Kafka consumer group "reprocess-v2"
     - Starts reading from offset 0 (beginning of log)
     - Writes to new serving store "results-v2"

  2. v1 and v2 run in parallel until v2 catches up to live

  3. Cut over queries from results-v1 to results-v2

  4. Shut down v1, drop results-v1

  Zero downtime, single codebase, no batch pipeline needed.

KAFKA AS THE IMMUTABLE LOG
────────────────────────────
  Kafka's role in Kappa Architecture:
  - Ordered, immutable, append-only: same properties as Lambda's HDFS master dataset
  - Replayable: consumers can seek to any offset
  - Partitioned: horizontal scale for high throughput ingestion

  Retention strategies:
  Default:         7-day retention (insufficient for full reprocess)
  Extended:        30–180 day retention (high disk cost at scale)
  Log compaction:  Retain latest value per key (good for entity state)
  Tiered Storage:  Kafka 3.6+ offloads old segments to S3/GCS; replays
                   from object storage on demand — virtually unlimited retention

  Kafka Tiered Storage example (kafka config):
  log.retention.ms = -1               # unlimited
  remote.log.storage.enable = true
  remote.log.storage.manager.class = org.apache.kafka.server.log.remote...

KAPPA VS LAMBDA COMPARISON
────────────────────────────
  Dimension              Lambda                    Kappa
  ─────────────────────  ────────────────────────  ──────────────────────────
  Code paths             2 (batch + stream)        1 (stream only)
  Consistency risk       High (drift between paths) Low
  Reprocessing mechanism Batch layer re-runs        Replay from Kafka offset 0
  Operational complexity High (2 systems)           Low (1 system)
  Full-history accuracy  High (batch is accurate)   High (if full retention)
  Arbitrary batch algo   Yes (Spark over HDFS)      Limited (stream state)
  Best for               Legacy batch + add stream  New builds, agile teams

Pros and cons

Pros

  • Single codebase — business logic is implemented once, in the stream processor; no batch/stream divergence.
  • Simpler infrastructure: one processing engine, one event log, one serving store.
  • Reprocessing with zero downtime: run new job in parallel, cut over when caught up, discard old.
  • Kafka Tiered Storage enables virtually unlimited log retention at low cost, removing the main historical objection to Kappa.

Cons

  • Reprocessing is slow: replaying months of data through a streaming pipeline takes time proportional to total data volume, not just recent data.
  • Stream processing has limited support for multi-pass algorithms (PageRank, iterative ML training) — batch Spark is required for these.
  • Kafka long retention at scale is expensive; Tiered Storage mitigates this but adds operational complexity.
  • Exactly-once semantics and stateful stream processing have a learning curve; incorrect checkpoint configuration leads to data loss or duplicates.

Implementation notes

Consumer group isolation: Each stream processing job (including reprocessing runs) must use a distinct Kafka consumer group. This allows v1 (live) and v2 (reprocessing) to read the same topic independently without interfering with each other's offsets. Never reuse consumer group IDs between different versions of a job.

Serving store versioning: During reprocessing, write to a separate serving store namespace (results_v2). Switch the query layer to the new store only after v2 has fully caught up with the live offset. This avoids serving stale or partially-reprocessed data to users.

Common failure modes

  • Insufficient Kafka retention: With default 7-day retention, a reprocessing job started after data has been purged cannot replay the full history — the architecture loses its core reprocessing capability.
  • No exactly-once semantics: Without idempotent producers, transactional writes, and proper offset management, reprocessing may produce duplicates in the serving store.

Decision checklist

  • Is Kafka (or equivalent replayable log) configured with sufficient retention for full history reprocessing?
  • Does each stream processing job use a distinct consumer group?
  • Is exactly-once semantics configured in the stream processor?
  • Is the serving store versioned so reprocessing and live results are isolated until cutover?
  • Are multi-pass algorithms (if any) handled separately by a batch pipeline rather than forced into streaming?

Example use cases

  • Real-time user activity feed: Events flow into Kafka → Flink computes per-user activity aggregates → results stored in Redis; when aggregation logic changes, deploy v2 Flink job, replay from offset 0, cut over Redis key prefix, discard v1.
  • Fraud score pipeline: Transaction events → Kafka (30-day retention) → Flink computes velocity features (transactions per 1h, 24h, 7d per card) → serving layer for real-time scoring; new model version triggers full reprocessing of 30 days of transactions.
  • Lambda Architecture — The pattern Kappa simplifies; useful when complex batch algorithms are genuinely required.
  • Spark Architecture — Spark Structured Streaming is a common stream processor in Kappa deployments; Spark batch is still used for multi-pass algorithms alongside Kappa.
  • Event Streaming — Kafka as the immutable log at the heart of Kappa Architecture.

Further reading