Reliability & Resilience Messaging

Poison Message Handling

Detecting and isolating messages that repeatedly cause processing failures to prevent queue blockage.

⏱ 8 min read

What it is

A poison message is a message that cannot be successfully processed regardless of how many times it is retried — not because of transient infrastructure issues, but because of an intrinsic problem with the message itself: a malformed payload that fails schema validation, a message referencing an entity that no longer exists, a message whose processing triggers a bug in the consumer code, or a message whose content violates a business rule that the consumer cannot satisfy. Unlike transient failures, poison messages will never succeed on retry; every attempt consumes resources and occupies queue capacity until the message is detected and isolated.

Poison message handling is the process of detecting when a message has reached its maximum delivery count without success, classifying the nature of the failure, routing the message to a quarantine destination (dead letter queue, parking lot topic, or poison queue), alerting the operations team, and providing a workflow for root cause analysis, potential fix, and safe replay. The key insight is that detection is based on delivery count threshold rather than trying to predict whether a specific message will succeed — any message that fails more than N times is treated as potentially poison.

Why it exists

Unhandled poison messages cause one of the most insidious queue failure modes: queue blockage. In ordered queues or single-consumer queues, a poison message at the head of the queue prevents all subsequent messages from being processed until the poison message is removed, causing a complete processing halt. Even in unordered queues, a poison message cycling through delivery and return consumes consumer capacity, inflates retry counters, and occupies queue depth metrics — masking the true backlog depth. The problem is self-reinforcing: the more the consumer retries the poison message, the more resources are consumed, and the less capacity is available for healthy messages.

When to use

  • Any message-driven system where messages can fail for reasons intrinsic to the message content (schema violations, business rule violations, missing referenced entities).
  • Ordered consumer patterns (Kafka partition ordering, SQS FIFO queues) where a blocked message halts all subsequent message processing.
  • Systems that process messages from multiple producers with different schemas or quality levels — schema drift is a common poison message source.
  • Long-running consumer pipelines where a single poison message could halt days of processing if not detected and isolated quickly.

When not to use

  • There is no practical scenario where poison message detection should be completely disabled — all message-driven systems benefit from delivery count tracking and DLQ routing.
  • However, the threshold should be adjusted based on whether failures are expected to be transient (high threshold: 10+) or mostly structural (low threshold: 3–5).

Typical architecture

Poison Message Detection and Quarantine Flow
════════════════════════════════════════════════

Message arrives at consumer
          │
          ▼
  ┌───────────────────┐
  │ Delivery Count    │
  │ check:            │
  │ count < maxRetries│ ──── attempt processing ───►
  │ count ≥ maxRetries│
  └────────┬──────────┘
           │ threshold exceeded
           ▼
  Classify failure type:
  ┌─────────────────────────────────────────────────┐
  │ Structural failure    │  Infrastructure failure  │
  │ (malformed payload,   │  (DB down, API timeout,  │
  │  missing reference,   │  external service error) │
  │  business rule fail)  │                          │
  │  → Poison message     │  → Transient — keep      │
  │  → Route to DLQ       │    retrying with backoff │
  └─────────────────────────────────────────────────┘
           │ poison
           ▼
  Write to Poison Queue / DLQ:
    - Original message body
    - Failure reason + stack trace
    - Delivery count
    - First failure timestamp
    - Consumer group / pod ID

           │
           ▼
  Alert operations team
  → Pagerduty / Slack: "Poison message in orders-dlq"
  → Link to diagnostic dashboard

Pros and cons

Pros

  • Prevents queue blockage — poison messages are quarantined rather than cycling indefinitely and blocking healthy message processing.
  • Preserves the original message body and failure context, enabling root cause analysis and safe replay after a fix.
  • Distinguishes structural failures from transient failures, enabling appropriate response (fix vs. wait).
  • DLQ alerts on poison messages provide early warning of producer-side schema changes or upstream data quality issues.
  • Forces the team to reason about failure modes and define explicit recovery procedures during design.

Cons

  • Requires operational process — teams must monitor the poison queue and act on messages rather than letting them accumulate.
  • Distinguishing poison messages from transient failures based solely on delivery count can be imprecise.
  • High maxReceiveCount means poison messages take many retries to detect, consuming consumer resources and introducing lag.
  • Replay after fix requires careful idempotency analysis — some messages may have been partially processed before the failure.

Implementation notes

Consumer code should catch exceptions, classify them (structural vs. transient), and either NACK (return to queue for retry) or explicitly forward to the DLQ. Do not rely solely on the broker's automatic dead-lettering by delivery count — add consumer-side logging of the failure reason, exception class, and message metadata to the DLQ message header. This context is critical for rapid diagnosis. For Kafka, the Spring Kafka SeekToCurrentErrorHandler and DeadLetterPublishingRecoverer provide a production-ready poison message implementation for partition-ordered consumers.

Tune maxReceiveCount (SQS) or equivalent based on the observed transient failure rate for the specific consumer. If the dependency is healthy 99.9% of the time and failures typically resolve in 1–2 retries, a threshold of 3–5 is appropriate. If the dependency can be down for extended periods and messages must not be lost, a higher threshold with longer visibility timeout may be appropriate — but combine this with a circuit breaker that pauses consumption entirely during outages rather than accumulating delivery count against healthy messages.

Common failure modes

  • Confusing poison with transient: setting maxReceiveCount=3 when the downstream is routinely down for 5+ minutes; messages go to DLQ after 3 quick failures that would have resolved on the 4th attempt.
  • No structured failure metadata: messages arrive in DLQ with no context about why they failed, requiring manual inspection of consumer logs to determine root cause.
  • Ordered queue halt: in FIFO queues or Kafka partitions, a poison message at offset N blocks all messages at N+1, N+2, etc. until the poison message is explicitly skipped or acknowledged.
  • Replay without fix: replaying DLQ messages without first fixing the bug that caused them to fail, sending them back to DLQ immediately.
  • No alerting on DLQ depth: poison messages accumulate silently for weeks before anyone notices the queue has grown to thousands of unprocessed messages.

Decision checklist

  • Does the consumer distinguish structural failures (NACK to DLQ) from transient failures (NACK to retry queue) at the exception level?
  • Is failure context (exception type, stack trace, delivery count, consumer ID) persisted to the poison queue message headers?
  • Is maxReceiveCount set based on actual observed transient failure duration rather than an arbitrary default?
  • Is there an alert on non-zero DLQ depth that pages the on-call engineer?
  • Is there a documented replay workflow that includes verifying the fix, checking idempotency, and monitoring secondary DLQ during replay?
  • For ordered consumers (Kafka, FIFO SQS), is there a mechanism to skip a poison message without losing offset progress?

Example use cases

  • An order event consumer fails to deserialise a message because the producer team added a required field without backward-compatible defaults — detected as poison after 3 attempts, quarantined to DLQ, producer team alerted, fix deployed, messages replayed.
  • A payment processing consumer encounters a message for a user account that was deleted — structural failure, routed to DLQ with context, operator manually verifies refund is not required, message discarded with audit log.
  • A Kafka consumer using Spring Kafka's DeadLetterPublishingRecoverer routes failed messages to a .DLT topic automatically after exhausting retries via FixedBackOffPolicy.

Further reading