Dead Letter Queues (DLQ)

What it is

A Dead Letter Queue (DLQ) is a secondary queue or topic that receives messages that could not be successfully processed by the primary consumer after a configured number of delivery attempts. Instead of silently discarding failed messages or blocking the queue indefinitely, the broker routes them to the DLQ, where they can be inspected by operators, diagnosed for root cause, and either replayed after a fix or intentionally discarded with an audit log entry. The DLQ preserves the failed message's body, attributes, and broker metadata, making it possible to understand exactly what failed and why.

The DLQ pattern is natively supported by most major message brokers: Amazon SQS has a RedrivePolicy with maxReceiveCount; RabbitMQ uses dead-lettering via exchange configuration; Azure Service Bus has a built-in dead-letter sub-queue; ActiveMQ routes to a DLQ.* destination automatically. Kafka does not have native DLQ support but the pattern is implemented by application code routing failed messages to a separate error topic.

Why it exists

Without a DLQ, a message that repeatedly fails processing has two bad outcomes: the broker retries it indefinitely (blocking forward progress and consuming resources), or the broker eventually discards it silently (causing silent data loss). The DLQ provides a third option — quarantine the failing message without blocking the queue and without losing it. This is essential for systems where message loss is unacceptable (financial transactions, order events, audit logs) and where the cause of failure may require investigation or a code fix before safe replay.

DLQs also serve as an operational signal: a non-empty DLQ is an alert that something is broken in the processing pipeline. Teams that monitor DLQ depth as a key metric catch processing failures within minutes rather than discovering them hours later through downstream data inconsistencies.

When to use

• Any message-driven system where message loss is unacceptable and failed messages need recovery capability.
• Pipelines where message processing failures may be caused by bugs that can be fixed and replayed.
• Systems with heterogeneous message formats where occasionally malformed messages need manual inspection.
• Event-driven architectures where the DLQ provides an audit trail of every failed event for compliance or debugging.
• Workflows with external dependencies that may be temporarily unavailable — messages that fail due to a downstream outage can be replayed after recovery.

When not to use

• When every message failure indicates a permanent, unrecoverable condition (e.g., message refers to a resource that has been permanently deleted) — log and discard instead.
• Extremely high-throughput, low-value streams where the cost of DLQ management exceeds the value of the messages (e.g., raw telemetry data where loss is acceptable).
• When there is no operational process to review and act on DLQ messages — an unmonitored DLQ provides false safety and will grow without bound.

Typical architecture

Dead Letter Queue Flow — Amazon SQS
═══════════════════════════════════════════════════════

Producer → Main Queue → Consumer
                │           │ processing fails
                │           │ (exception thrown)
                │           ↓ message visibility timeout expires
                │       Message returned to queue
                │           │
                │       Attempt 2 → fails
                │       Attempt 3 → fails
                │           │ maxReceiveCount (3) exceeded
                │           ▼
                │       Dead Letter Queue
                │           │
                │    ┌──────┴──────────────────────┐
                │    │ DLQ Processing Options:       │
                │    │  1. CloudWatch alarm on depth │
                │    │  2. DLQ consumer for analysis │
                │    │  3. Manual inspection tool    │
                │    │  4. Replay after code fix     │
                │    │  5. Discard + structured log  │
                │    └──────────────────────────────┘
                │
SQS RedrivePolicy config:
  {
    "deadLetterTargetArn": "arn:aws:sqs:...:MyQueue-DLQ",
    "maxReceiveCount": 3
  }

Pros and cons

Implementation notes

Name DLQs consistently: use the convention {source-queue-name}-DLQ or {source-queue-name}-dlq so the relationship between source and DLQ is immediately obvious. For SQS, set maxReceiveCount based on the type of failure you're protecting against: 3 attempts is appropriate for transient failures; higher counts (5–10) if you expect occasional external dependency hiccups that resolve within a few attempts. Set the DLQ's message retention period long enough to allow investigation — 14 days is the SQS maximum and a reasonable default.

Implement a CloudWatch alarm (or equivalent) on ApproximateNumberOfMessagesVisible for the DLQ, alerting on any message count greater than 0. A DLQ should be empty in steady state — any depth is an anomaly. Build a replay tool that reads from the DLQ and re-publishes to the source queue with a rate limiter, so that a large backlog can be replayed safely without overwhelming the consumer. Before replaying, ensure the fix has been deployed, verify the replay is idempotent, and monitor for secondary DLQ depth during replay.

Common failure modes

• Unmonitored DLQ: messages accumulate silently for days or weeks until a downstream data inconsistency reveals the problem.
• maxReceiveCount too high: a genuinely poison message gets retried 50 times before routing to DLQ, consuming excessive consumer resources and delaying processing of healthy messages.
• No replay tooling: messages are visible in the DLQ but there is no mechanism to replay them after a fix, requiring manual re-injection of each message.
• Non-idempotent replay: replaying a message that partially succeeded on the first attempt causes duplicate payments, duplicate inserts, or duplicate notifications.
• DLQ retention too short: messages expire from the DLQ before the team investigates the failure, causing data loss despite the DLQ existing.

Decision checklist

• Is every source queue paired with a DLQ, and is the naming convention consistent?
• Is a CloudWatch alarm (or equivalent) set to alert on any non-zero DLQ depth?
• Is maxReceiveCount tuned to the expected transient failure rate (typically 3–5, not >10)?
• Is there a documented and tested replay process for re-processing DLQ messages after a fix?
• Is replay designed to be idempotent to prevent duplicate side effects?
• Is the DLQ message retention period long enough for the team's typical incident investigation cycle?
• Are DLQ messages enriched with metadata (original failure reason, timestamp, attempt count) to facilitate diagnosis?

Example use cases

• An SQS-based order processing pipeline routes orders that fail to deserialise (due to a schema change) to a DLQ, where they are held until the consumer is updated and then replayed successfully.
• A payment event consumer that fails when the payment gateway is down routes events to a DLQ; after the gateway recovers, all events are replayed with idempotency keys preventing duplicate charges.
• An Azure Service Bus-based notification system where messages for deleted accounts accumulate in the DLQ and are periodically reviewed and discarded with an audit log.

Related patterns

• Poison Message Handling — the specific category of DLQ messages that fail due to content rather than transient infrastructure.
• At-Least-Once Delivery — the delivery model that necessitates DLQs for repeated-failure handling.
• Idempotency — the safety property that makes DLQ replay safe.

Further reading

• Amazon SQS Dead-Letter Queues — AWS Documentation — configuration and redrive policy reference.
• Dead-letter queues — Azure Service Bus — built-in DLQ sub-queue and dead-lettering conditions.