Communication PatternsAsyncStreaming

Event Streaming

A continuous, durable, replayable log of ordered events — Kafka's architecture, exactly-once semantics, stream-table duality, and how streaming differs from traditional queuing.

⏱ 14 min read

What it is

Event streaming treats events as an immutable, ordered log that any consumer can read at any offset, at any time, as long as the data is within the retention window. Apache Kafka is the canonical implementation: events are appended to partitions within a topic; partitions are distributed across broker nodes for parallelism and replicated across nodes for fault tolerance. A consumer group is a logical subscriber — each partition is assigned to exactly one member of the group, giving both fan-out (multiple groups read the same topic independently) and parallelism (partitions within a group are processed concurrently). Every consumer tracks its position via an offset — a monotonically increasing integer per partition — so it can replay from any historical point or resume from where it left off.

This model differs fundamentally from queues: data is not deleted on consumption; it is retained for a configurable period (hours, days, or indefinitely with log compaction). This makes Kafka both a message broker and a durable event store, enabling event replay, new consumer bootstrapping, and audit trails without a separate archival system.

Why it exists

Traditional message queues delete messages after consumption. This is fine for task dispatch, but leaves no history. Systems needing to rebuild state (new microservice bootstrapping, recovering from data corruption, training ML models on historical events) require the raw event history. Kafka's log-centric design was born at LinkedIn to decouple data pipelines — activity feeds, metrics, log aggregation — all running off the same durable event backbone. The result is a single source of truth for system events that every downstream system can consume independently, at its own pace, and replay at will.

When to use

Event sourcing or CQRS write-side store — events are the primary source of truth; projections are derived by replaying them.
Multiple independent consumers — analytics, search indexing, caching, ML feature pipelines all consuming the same event stream without interference.
Audit trail requirements — financial transactions, compliance events, user activity logs needing permanent, tamper-evident history.
Real-time stream processing — computing rolling aggregates, detecting anomalies, joining streams with tables.
Cross-datacenter replication — MirrorMaker 2 (or Confluent Replicator) replicates topics across regions for disaster recovery or geo-distributed processing.

When not to use

Simple task queues — Kafka's operational overhead (ZooKeeper/KRaft, partition rebalancing, offset management) is excessive for basic work queues; SQS or RabbitMQ is simpler.
Very small scale — Kafka's minimum cluster footprint (3 brokers for production replication) makes it disproportionate for low-volume systems.
Request-response semantics — Kafka is strictly one-way pub/sub; implementing RPC over Kafka is an anti-pattern.
When exactly-once is business-critical without careful tuning — exactly-once semantics in Kafka require idempotent producers + transactional APIs; misconfiguration silently falls back to at-least-once.

Typical architecture

A topic is split into partitions, each an ordered, append-only log. Producers write to a partition determined by the message key's hash. Each partition is replicated to N brokers (replication factor 3 is standard). Consumer groups each maintain an independent offset cursor per partition, stored in the internal __consumer_offsets topic. A Kafka Streams application reads input topics, performs stateful computations (joins, aggregations using RocksDB state stores), and writes results to output topics — all within a single JVM process.

Producers (keyed by order_id)
  │
  ▼
Topic: "orders"   (replication-factor=3, partitions=12)
  ├── Partition 0  [offset 0..5423]  ──► Broker 1 (leader)
  ├── Partition 1  [offset 0..4891]  ──► Broker 2 (leader)
  └── Partition 2  [offset 0..6102]  ──► Broker 3 (leader)
        │
  ┌─────┼────────────────────┐
  │     │                    │
CG:analytics    CG:inventory    CG:fraud
(offset=5423)   (offset=3000)   (offset=6102)
  │                  │               │
[Spark/Flink]  [Kafka Streams]  [Flink job]

Pros and cons

Pros

Replayability — replay from any offset for new consumers, bug fixes, or backfills.
High throughput — sequential disk I/O and batched sends enable millions of events/sec on modest hardware.
Independent consumers — multiple consumer groups each receive all events; no fan-out plumbing required.
Durable event store — events persist for days or indefinitely, serving as both bus and archive.
Log compaction — for change-log topics, retain only the latest value per key, bounding disk usage while preserving full state.

Cons

Operational complexity — cluster management, partition rebalancing, retention policy tuning, and schema registry require dedicated expertise.
Partition count is fixed at creation (increasing requires manual reassignment and temporary performance impact).
Consumer group rebalancing causes brief processing pauses when members join or leave.
Exactly-once semantics require transactional producers and careful configuration — the default is at-least-once.
Not suitable for long-retention random access at scale — Kafka is a log, not a queryable database.

Implementation notes

Partition count: Choose initial partition count generously — you cannot easily decrease it later, and increasing it requires partition reassignment. Rule of thumb: target 1 partition per consumer instance at peak load, with headroom for growth. Common starting points: 12, 24, or 48 partitions for most topics.

Log compaction: Enable compaction (cleanup.policy=compact) for changelog topics (entity state changes keyed by entity ID). Kafka retains only the latest message per key, bounding storage. A null-value (tombstone) message deletes the key from the compacted log. Compaction runs in the background and does not affect producers or consumers.

Exactly-once semantics (EOS): Requires enable.idempotence=true on the producer (prevents duplicates from retries) and transactional APIs (initTransactions(), beginTransaction(), commitTransaction()) when producing to multiple partitions or topics atomically. Consumer-side EOS requires reading with isolation.level=read_committed to skip uncommitted transactional messages.

Kafka Streams vs Flink: Kafka Streams is a Java library that runs in-process — no separate cluster, simple deployment, ideal for straightforward enrichment, filtering, and aggregation. Apache Flink is a full stream processing framework supporting complex event processing, stateful joins across streams, watermarking for event-time semantics, and large-scale analytics — at the cost of a separate Flink cluster to manage.

Consumer lag monitoring: Track consumer_lag (end offset − consumer offset) per partition per consumer group. Alert when lag exceeds a threshold relative to the topic's throughput rate. Use Burrow, Cruise Control, or Confluent Control Center. Lag growth indicates consumers are falling behind the produce rate — a leading indicator of SLA breach.

Common failure modes

Uncontrolled partition rebalancing: Frequent restarts or deployments trigger rebalancing storms, pausing all consumers in the group. Mitigate with static membership (group.instance.id) and incremental cooperative rebalancing.
Under-replicated partitions: A broker falling behind replication creates under-replicated partitions, risking data loss if the leader fails. Monitor UnderReplicatedPartitions metric; alert at > 0.
Log retention expiry before consumer catch-up: If a consumer falls too far behind and retention expires, it loses messages permanently. Use long retention or compact+delete policy for critical topics.
Schema evolution breaking consumers: A producer adds a required field without a compatible schema; consumers on old schema versions crash on deserialisation. Use a schema registry with BACKWARD compatibility enforcement.
Hot partition due to poor key selection: All messages land on one partition because a low-cardinality key was chosen (e.g., all messages keyed as "global"), creating a throughput bottleneck.

Decision checklist

Do consumers need to replay historical events? If yes, Kafka's durable log is a strong fit.
Are there multiple independent consumer groups needing the same events? Kafka's consumer group model handles this natively.
Is the partition key chosen to distribute load evenly and preserve ordering requirements?
Is a schema registry deployed with compatibility rules enforced on topic schemas?
Is consumer lag monitored per group per partition, with alerts configured?
Is the replication factor ≥ 3 and min.insync.replicas ≥ 2 for durability?
For exactly-once requirements: are idempotent producers and transactional APIs enabled and tested?

Example use cases

Event sourcing backbone: All domain events (order placed, payment captured, shipment dispatched) written to Kafka; microservices rebuild their read models by replaying the event log from offset 0.
Real-time fraud detection: Transaction events stream through a Kafka Streams topology that joins with account risk profiles and emits alerts for suspicious patterns within 50 ms.
Data warehouse CDC pipeline: A Debezium connector captures database change events, writes them to Kafka, and a Flink job transforms and loads them into Snowflake for analytics — providing sub-minute data freshness.

Publish/Subscribe — simpler, lighter-weight variant without log retention and replay.
Transactional Outbox — reliable way to write domain events into Kafka from a database transaction.
Competing Consumers — parallelism within a consumer group across partitions.
Event Sourcing — architectural pattern that uses a streaming log as the primary data store.