Communication PatternsMessagingScalability

Competing Consumers

Horizontal scaling of message processing by deploying multiple identical workers that race to lease and process messages from a shared queue — the fundamental scaling primitive for async workloads.

⏱ 10 min read

What it is

Competing consumers is the pattern where multiple identical consumer instances all listen on the same queue. When a message arrives, one consumer leases it (making it invisible to others), processes it, and acknowledges it — the broker ensures only one consumer processes each message. Adding more consumer instances increases throughput linearly, since each new instance handles additional messages in parallel. This is the horizontal scaling primitive for queue-based workloads: scale out when queue depth grows, scale in when the queue is empty.

The pattern differs from a Kafka consumer group: in Kafka, each partition is assigned to exactly one consumer in the group, and scaling beyond the partition count yields no benefit. In a traditional queue (SQS, RabbitMQ), any number of competing consumers can be active simultaneously, and each pulls work at its own pace. This makes competing consumers on queues suitable for heterogeneous worker pools and variable message processing times, while Kafka consumer groups are better suited to ordered, high-throughput streaming.

Why it exists

A single consumer is a bottleneck: it processes one message at a time, and a backlog accumulates if the produce rate exceeds its processing rate. Competing consumers allow the system to scale horizontally until the queue drains at the desired rate. The queue acts as a natural backpressure buffer, and the consumer pool size becomes an autoscaling parameter rather than a configuration constant. This decouples throughput from the processing speed of any individual consumer instance.

When to use

Variable or bursty workloads — spin up extra consumers during peak, scale to zero at night.
CPU/IO-intensive processing — parallelise image transcoding, PDF rendering, or ML inference across a pool of specialised workers.
Processing order is irrelevant — if messages are independent and interchangeable, any consumer can handle any message.
Fault-tolerant task execution — a crashing consumer does not lose its in-flight message; the visibility timeout expires and another consumer picks it up.
SLA-driven autoscaling — maintain a target queue depth (e.g., <100 messages) by scaling the consumer pool.

When not to use

When processing order matters — multiple consumers inherently process messages out of order; use a single consumer, partitioned queues, or ordered Kafka partitions instead.
When messages for the same entity must be serialised — two consumers processing two updates to the same order concurrently can cause lost updates. Partition by entity key or use ordered single-consumer lanes.
When fan-out is needed — competing consumers ensure each message is handled by exactly one consumer; if all consumers need every message, use pub/sub.

Typical architecture

N consumer instances poll the queue. Each consumer fetches a batch of messages (prefetch/batch size), marks them invisible, processes them, and ACKs each one. Autoscaling monitors queue depth or consumer lag; when depth exceeds a threshold, the autoscaler adds instances. KEDA (Kubernetes Event-Driven Autoscaling) enables scale-to-zero when the queue is empty.

Queue (SQS / RabbitMQ)
  │  Depth: 5,000 messages
  │
  ├─ Consumer-1  [prefetch=10] → process → ACK
  ├─ Consumer-2  [prefetch=10] → process → ACK
  ├─ Consumer-3  [prefetch=10] → process → ACK
  └─ Consumer-4  [prefetch=10] → process → crash
                                  ↓
                           visibility timeout expires
                                  ↓
                           message reappears in queue
                                  ↓
  └─ Consumer-1 or 2 picks it up again

Autoscaler watches:
  queue depth > 500  → scale out (+2 instances)
  queue depth < 50   → scale in  (-1 instance)
  queue depth = 0    → scale to zero (KEDA)

Pros and cons

Pros

Linear throughput scaling — double the consumer count to roughly double the processing rate.
Fault tolerant — consumer crashes do not lose messages; the visibility timeout re-delivers.
Simple autoscaling — queue depth is a clean, observable signal for scaling decisions.
Consumers are stateless and interchangeable — easy to deploy, restart, and replace.

Cons

No ordering guarantee — messages are processed in the order consumers happen to claim them, not the order they were enqueued.
Duplicate processing risk — if a consumer crashes after processing but before ACKing, the message is redelivered to another consumer.
Prefetch misconfiguration can unbalance load — a high prefetch count pins many messages to a slow consumer while fast consumers are idle.
Entity-level ordering requires extra design — partitioning by entity key or per-entity queues adds complexity.

Implementation notes

Prefetch count tuning: In RabbitMQ, basic.qos(prefetch_count) limits how many unacknowledged messages a consumer holds. Too high: messages are pinned to slow consumers and cannot be picked up by idle ones. Too low: frequent broker round-trips reduce throughput. Rule of thumb: start with prefetch = 2× the number of threads per consumer. In SQS, MaxNumberOfMessages in ReceiveMessage serves a similar purpose.

Graceful shutdown: Consumers should trap SIGTERM, stop fetching new messages, and wait for in-flight messages to finish processing and ACK before exiting. This prevents the visibility timeout from expiring on a message that was nearly done, causing redundant re-processing.

Consumer groups in Kafka vs competing consumers: In Kafka, a consumer group assigns partitions to group members — at most one member per partition processes a given partition. This is ordered within a partition but limits parallelism to the partition count. In a traditional queue, the number of competing consumers is unbounded by partition count. For Kafka: if you need more parallelism than partitions, add partitions at creation time (not after). For SQS/RabbitMQ: scale consumers freely.

Scaling metrics: Use both queue depth (backlog) and consumer lag rate (is the backlog growing or shrinking?). A shrinking backlog at high queue depth is fine — consumers are catching up. A growing backlog means consumers are undersized. Set scale-out triggers at sustained queue depth growth for 60 seconds (avoids scaling on transient bursts).

Ordered processing with partitioning: For messages that must be processed in order per entity (all updates to order-ID-123 must be sequential), partition messages by entity key into per-entity queues (SQS FIFO with MessageGroupId) or Kafka partitions. Assign one consumer per partition to preserve order while still parallelising across entities.

Common failure modes

Consumer starvation under high prefetch: One slow consumer holds a large prefetch batch while fast consumers sit idle, reducing effective throughput below the theoretical maximum.
Visibility timeout too short: Fast consumers are fine, but a slow consumer's visibility timeout expires before processing completes, causing re-delivery to another consumer — duplicate processing.
Non-idempotent processing: A message delivered twice (after crash or timeout) triggers double billing, double email send, etc., because the consumer does not check if it already processed this message.
Scale-out without limit: Autoscaler scales to hundreds of consumers overwhelming the downstream database or API; always set a maximum consumer count and rate-limit downstream calls.

Decision checklist

Is message processing order irrelevant (or handled via partitioning by entity key)?
Is each consumer idempotent — safe to process the same message twice?
Is the visibility timeout set to at least 2× the maximum expected processing time?
Is prefetch count tuned relative to the number of processing threads per consumer?
Is graceful shutdown implemented so in-flight messages are ACKed before consumer termination?
Is autoscaling bounded by a maximum consumer count to protect downstream systems?
Is consumer lag monitored with alerts so backlog growth is detected proactively?

Example use cases

Email dispatch pool: 50 consumer instances drain an email queue; at night KEDA scales to zero; at 9am launch, autoscaler spins up 20 instances to handle the burst.
ML inference workers: GPU-equipped consumer instances process image classification requests from a queue; each processes one batch at a time and ACKs on completion.
Order fulfilment pipeline: Consumer pool processes order.confirmed messages; each consumer fetches order details, reserves inventory, and fires fulfilment — safe because each order ID appears once in the queue.

Message Queues — the underlying infrastructure for competing consumers.
Fan-Out / Fan-In — distribute work across workers and aggregate results (complementary, not the same).
Inbox Pattern — make competing consumers idempotent by deduplicating at the database level.