Communication PatternsAsyncEssential

Message Queues

A durable buffer between producers and consumers that decouples throughput, absorbs bursts, and guarantees at-least-once delivery — the backbone of reliable asynchronous task processing.

⏱ 11 min read

What it is

A message queue is a durable, ordered buffer that accepts messages from one or more producers and delivers each message to exactly one consumer (point-to-point). Producers enqueue messages without knowing which consumer will handle them or when. Consumers dequeue — or, more precisely, lease — messages, process them, then acknowledge completion. If the consumer fails before acknowledging, the broker re-queues the message after a configurable visibility timeout expires, ensuring no work is silently lost. This at-least-once guarantee means duplicate delivery is possible — consumers must be idempotent.

Queues differ from pub/sub topics in fan-out semantics: a message in a queue is handled by exactly one consumer instance, making queues ideal for task distribution and load balancing. A message in a pub/sub topic is replicated to every subscription. In practice many architectures combine both: a pub/sub topic fans out to multiple queues, each processed independently by a worker pool.

Major implementations include Amazon SQS (fully managed, unlimited scale), RabbitMQ (AMQP, rich routing, acks at transport level), and Apache ActiveMQ / ActiveMQ Artemis (JMS-compatible, used heavily in Java enterprise environments). Redis Streams offer queue-like semantics with consumer groups for lightweight deployments.

Why it exists

Without a queue, a producer must either call the consumer synchronously (tight coupling, cascading failures) or accept message loss if the consumer is unavailable. A queue absorbs bursts — if 10,000 messages arrive in a second but the consumer pool handles 1,000/s, the queue buffers the excess and drains it over time, protecting downstream services from overload. This temporal decoupling is especially valuable for spike traffic (flash sales, batch imports) and for isolating slow, resource-intensive processing (PDF rendering, video transcoding) from fast request-serving services.

When to use

  • Asynchronous task offloading — sending emails, generating reports, processing images: tasks where the user does not wait for the result.
  • Rate limiting downstream processing — produce at whatever rate events arrive; consume at a sustainable rate for the downstream service.
  • Reliable task delivery with at-least-once guarantee — the message persists in the queue even if the consumer crashes mid-processing.
  • Work distribution across a pool of consumers — multiple workers consume the same queue; the broker load-balances naturally.
  • Decoupling microservices across deployment boundaries — one team owns the producer, another the consumer, with the queue as the contract boundary.

When not to use

  • Fan-out to multiple independent services — use a pub/sub topic instead; a single queue with multiple consumers competes for messages rather than each getting a copy.
  • Ordered processing across all messages globally — queues parallelise consumption; global order requires a single consumer, which eliminates horizontal scaling. Use a Kafka partition instead.
  • Sub-millisecond latency — queue serialisation and broker round-trips add overhead incompatible with latency-critical paths.
  • Event replay or audit log — standard queues delete messages after acknowledgment; use an event log (Kafka) if replay is required.

Typical architecture

The producer writes a message to the queue (enqueue). The broker persists it to durable storage. A consumer polls (receives) the message — the broker hides it from other consumers for the visibility timeout. The consumer processes and acknowledges (deletes) the message. If processing fails or the consumer crashes, the visibility timeout expires and the message becomes visible again for another consumer. After maxReceiveCount failures, the message is routed to a dead letter queue (DLQ) for manual inspection.

Producer ──► Queue (durable)
                │
    ┌───────────┼───────────┐
    │           │           │
Consumer-1  Consumer-2  Consumer-3
(lease msg, (idle)      (lease msg,
 process,               process,
 ACK)                   crash → visibility timeout
                         → message reappears)

After maxReceiveCount failures:
Queue ──► Dead Letter Queue (DLQ)
             │
          Alert / manual review

Pros and cons

Pros

  • Durability — messages survive producer and consumer restarts.
  • Load levelling — buffers burst traffic; consumers process at a stable rate.
  • Horizontal scaling — add consumers to drain backlog; remove them when idle.
  • Fault tolerance — visibility timeout + DLQ ensures no silent message loss.
  • Decoupled deployments — producer and consumer can be released independently.

Cons

  • At-least-once delivery — duplicate processing is possible; consumers must handle it.
  • No built-in replay — messages are deleted after ACK; not suitable as an event log.
  • FIFO is expensive — strict ordering requires a single-lane FIFO queue which limits throughput (SQS FIFO: 300 TPS per message group without batching).
  • Increased operational complexity vs. direct calls — queue depth, DLQ size, and consumer lag all need monitoring.
  • Latency floor — a queue adds at least one broker round-trip; not suitable for real-time interactions.

Implementation notes

Visibility timeout: Set to 1.5–2× the expected maximum processing time. Too short causes duplicates; too long delays re-processing after a consumer crash. Consumers should extend (heartbeat) the visibility timeout for long-running jobs.

Message acknowledgment: Always acknowledge after successful processing, not before. Some frameworks offer auto-ack on receipt — disable it for reliable processing. In RabbitMQ use basic.ack on success and basic.nack with requeue=false on unrecoverable failure to route to DLQ.

Dead letter queues: Configure a DLQ on every queue. Set an appropriate maxReceiveCount (typically 3–5). Alert on DLQ depth > 0 — every message in the DLQ represents a processing failure requiring investigation. Include enough context in the message payload (correlation ID, original timestamp, source) to diagnose failures.

Back-pressure: Monitor queue depth as a scaling signal. Use an autoscaler (KEDA, AWS Application Auto Scaling) to add consumer instances when depth exceeds a threshold. Define a maximum queue depth at which you stop accepting new messages or return 429 to producers.

FIFO vs standard queues: Standard queues offer best-effort ordering and nearly unlimited throughput. FIFO queues guarantee exactly-once processing and strict order within a message group, but at lower throughput. Use FIFO only when ordering is a hard requirement and throughput fits within limits.

Common failure modes

  • Poison message: A malformed message repeatedly fails processing, exhausts its retry count, and lands in the DLQ — but if the DLQ is not monitored, it silently accumulates.
  • Visibility timeout too short: Slow consumers process the message successfully but the broker re-delivers it before the ACK arrives, causing duplicate processing.
  • Consumer starvation: A large batch of high-priority messages blocks a fixed-size consumer pool; lower-priority messages wait indefinitely (solution: priority queues or dedicated pools).
  • No DLQ configured: Failed messages vanish after max retries, leading to silent data loss — the worst outcome in a reliable system.
  • Queue unbounded growth: Consumers are too slow or stopped; queue depth grows without an alarm, causing storage exhaustion or cascading producer failures.

Decision checklist

  • Does each message need to be processed by exactly one consumer? If multiple consumers need a copy, use a pub/sub topic.
  • Is the consumer idempotent? At-least-once delivery is default for all major queues.
  • Is the visibility timeout set to at least 1.5× the maximum expected processing time?
  • Is a DLQ configured, and is DLQ depth monitored with an alert?
  • Is queue depth exposed as a scaling metric to trigger autoscaling?
  • Are FIFO semantics required? If yes, does the expected throughput fit within the queue's FIFO limits?
  • Is the maximum message size within the broker's limit (SQS: 256 KB; if larger, use the Claim Check pattern)?

Example use cases

  • Order fulfilment: An e-commerce checkout service enqueues order.confirmed messages; a warehouse service processes them at capacity, picking and packing without blocking checkout.
  • Email/SMS dispatch: A notification queue buffers send requests; a worker pool calls the email provider API, retrying on transient failures without affecting the application.
  • Video transcoding pipeline: An upload service enqueues video IDs; a GPU worker pool picks up jobs, transcodes to multiple resolutions, and acknowledges on completion.
  • Publish/Subscribe — fan-out variant; one message delivered to multiple independent consumers.
  • Competing Consumers — the scaling pattern for worker pools consuming a shared queue.
  • Claim Check — handle payloads larger than the queue's message size limit.
  • Transactional Outbox — reliably enqueue messages atomically with a database write.

Further reading