Communication PatternsAsyncEssential

Publish/Subscribe (Pub/Sub)

An asynchronous messaging pattern in which publishers emit events to named topics and any number of subscribers receive their own copy — decoupling producers from consumers in space, time, and number.

⏱ 12 min read

What it is

Publish/Subscribe (Pub/Sub) is an asynchronous messaging model where message producers (publishers) send events to a named channel (a topic) without knowing who will consume them, and message consumers (subscribers) declare interest in a topic and receive all matching messages. The broker — Kafka, Google Pub/Sub, AWS SNS, Redis Pub/Sub, or NATS — sits between producers and consumers and is responsible for routing, storing (optionally), and delivering messages. This model removes the direct coupling present in request-response: the publisher does not know how many subscribers exist, does not wait for any acknowledgment, and continues publishing regardless of whether consumers are online.

The critical distinction between a topic (pub/sub) and a queue (point-to-point) is fan-out: a message published to a topic is independently delivered to every subscription, whereas a message in a queue is consumed by exactly one worker. In practice, systems often combine both — a topic fans out to multiple subscriptions, each of which is backed by a queue consumed by a pool of workers.

Why it exists

Distributed systems need to propagate state changes across service boundaries without tight coupling. Before pub/sub, services communicated via direct HTTP calls — adding a new consumer required changing the producer, and a slow consumer could block the producer. Pub/Sub removes this coupling entirely: the producer publishes once and the broker handles delivery to all interested parties. This enables independent deployability, autonomous team ownership, and organic addition of new consumers without producer changes. It also provides temporal decoupling — a subscriber can be offline and catch up on missed messages when it reconnects, provided the topic retains messages.

When to use

  • Broadcasting events to multiple downstream systems — an order.placed event needs to trigger inventory reservation, notification dispatch, analytics ingestion, and fraud scoring simultaneously.
  • Decoupling microservices — the order service should not know about the notification service; it simply publishes order.placed and any interested party can subscribe.
  • Real-time feed distribution — pushing social feed updates, market data ticks, or IoT sensor readings to thousands of subscribers.
  • Event-driven workflows — each stage of a workflow subscribes to the output topic of the previous stage, forming a pipeline without direct wiring.
  • Cache invalidation fanout — publishing a product.updated event so every cache layer can evict stale entries.

When not to use

  • When you need a direct reply — pub/sub is one-way; for request-response semantics use HTTP/gRPC or the async request-reply pattern (reply-to topic).
  • Simple point-to-point task delegation — if only one consumer should ever process a message, a queue gives you load balancing and simpler semantics without multi-subscriber confusion.
  • Low-volume, latency-sensitive operations — a pub/sub broker adds serialisation and network round-trip overhead that makes it inappropriate for sub-millisecond interactions.
  • When ordering across all partitions is required — most pub/sub systems guarantee order only within a partition; global total order requires a single partition, which limits throughput.

Typical architecture

Publishers write to a topic. The broker partitions the topic across multiple nodes (in Kafka, these are partition leaders). Each subscriber group maintains its own offset cursor into each partition. Pull-based consumers (Kafka, Google Pub/Sub) poll the broker; push-based consumers (AWS SNS, Redis Pub/Sub) receive messages as HTTP/WebSocket pushes. Subscription filters allow a subscriber to receive only a subset of topic messages matching a predicate (e.g., event_type = "order.placed").

Publisher A ──┐
Publisher B ──┼──► Topic: "orders"
Publisher C ──┘    ├── Partition 0 (key hash 0–3)
                   ├── Partition 1 (key hash 4–7)
                   └── Partition 2 (key hash 8–11)
                         │
          ┌──────────────┼──────────────┐
          │              │              │
    Subscription    Subscription   Subscription
    "notifications" "analytics"    "fraud"
    (filter: all)   (filter: all)  (filter: amount > 500)
          │              │              │
      [workers]      [workers]      [workers]
      (pull, ack)    (pull, ack)    (pull, ack)

Pros and cons

Pros

  • Producers and consumers are decoupled — add new subscribers without touching publishers.
  • Fan-out delivery — a single publish reaches N independent consumers.
  • Temporal decoupling — durable subscriptions survive consumer restarts and catch up from retained messages.
  • Natural backpressure — consumers pull at their own rate; the broker buffers bursts.
  • Replayability (with retention) — re-process historical events for new consumers or bug fixes.

Cons

  • No built-in reply — request-response requires awkward reply-to topic patterns.
  • Ordering is partition-scoped, not global — cross-partition ordering requires application-level sequencing.
  • Duplicate delivery is expected (at-least-once) — consumers must be idempotent.
  • Schema drift — producers and consumers evolve independently; schema registry and compatibility checks are essential.
  • Operational complexity — managing topics, partition counts, retention policies, and consumer lag requires dedicated tooling.

Implementation notes

Topics vs Queues: Use a topic when multiple independent consumers need the same event. Use a queue when you want competing consumers sharing work. Many cloud services combine both: AWS SNS (topic) fans out to multiple SQS queues (each with its own pool of workers).

Push vs Pull: Pull consumers (Kafka, Google Pub/Sub Streaming Pull) control their own rate, making backpressure natural. Push delivery (SNS HTTP endpoints, Redis Pub/Sub) reduces latency but requires consumers to handle burst and may overwhelm slow consumers. Prefer pull for high-throughput processing pipelines; push for near-real-time low-volume notifications.

Partition key selection: Choose a partition key that distributes load evenly and groups events that must be processed in order. A common choice for order events is order_id — all events for the same order land on the same partition, guaranteeing order. Avoid low-cardinality keys (e.g., event type) that create hot partitions.

Durable vs Ephemeral subscriptions: A durable subscription stores unacknowledged messages, allowing a consumer to reconnect and catch up. An ephemeral subscription loses messages when the consumer disconnects (Redis Pub/Sub). Use durable subscriptions for business-critical events; ephemeral only for real-time dashboards where missing a message is acceptable.

Subscription filters: Server-side filtering (SNS filter policies, Google Pub/Sub message filters) reduces unnecessary delivery and consumer-side processing. Define filters based on message attributes, not payload content, for efficiency.

Common failure modes

  • Consumer lag growth: A slow or crashing consumer group falls behind; if retention expires before the consumer catches up, messages are permanently lost.
  • Hot partition: A skewed partition key routes all traffic to one partition, overwhelming a single broker node and consumer worker.
  • Missing idempotency: At-least-once delivery causes duplicate processing — a charge runs twice, an email is sent twice — because consumers do not deduplicate.
  • Schema incompatibility: A producer changes a field type without a forward-compatible schema evolution; consumers break on deserialisation.
  • Dangling subscriptions: Subscriptions for decommissioned services accumulate unacked messages, growing storage and alerting on false consumer lag.

Decision checklist

  • Does more than one service need to react to this event? If yes, pub/sub is appropriate.
  • Is the event business-critical? Use a durable subscription with message retention.
  • Are consumers idempotent? At-least-once delivery is the default; design consumers accordingly.
  • Is a schema registry in place? Enforce compatibility (BACKWARD or FULL) to prevent breaking consumers.
  • Is the partition key chosen to avoid hot partitions while preserving necessary ordering?
  • Is consumer lag monitored with alerts for SLA-affecting thresholds?
  • Are dangling subscriptions and retention policies reviewed regularly?

Example use cases

  • Order processing pipeline: order.placed topic fans out to inventory, notification, billing, and analytics services — each consuming independently at their own pace.
  • Real-time market data: A pricing service publishes price ticks to a Kafka topic with symbol as partition key; trading algorithms and dashboards subscribe to specific symbols.
  • IoT telemetry ingestion: Millions of devices publish sensor readings to Google Pub/Sub; a stream processing pipeline subscribes, aggregates, and writes to a time-series database.
  • Message Queues — point-to-point counterpart; combine with topics for fan-out + work distribution.
  • Event Streaming — Kafka-style durable, replayable topic log with richer stream processing semantics.
  • Competing Consumers — scale consumers horizontally when a subscription has heavy processing load.
  • Event Notification — a lighter-weight variant focused on minimal payloads and follow-up fetches.

Further reading