Inbox Pattern

What it is

The inbox pattern is the consumer-side complement to the transactional outbox pattern. It solves the duplicate message delivery problem: message brokers guarantee at-least-once delivery, which means a consumer can receive the same message more than once (after a broker retry, after a consumer crash that leaves a message unacknowledged, or after the visibility timeout expires). If the consumer is not idempotent, duplicate delivery causes duplicate side-effects — double charges, double emails, double inventory decrements.

The inbox pattern prevents this by inserting the message ID into a dedicated inbox (or processed_messages) table in the same database transaction as the business side-effect. If the same message is received again, the consumer tries to insert the message ID but hits a primary key conflict — the message is detected as a duplicate and skipped without re-applying the side-effect. The uniqueness constraint in the database is the deduplication gate.

Why it exists

Building truly idempotent business logic is harder than it seems. Natural idempotency (e.g., setting a field to a value is idempotent by nature) works for some operations, but others — charging a payment, sending an email, decrementing a counter — are inherently non-idempotent. The inbox pattern provides a universal, mechanical way to make any consumer idempotent without requiring the business logic itself to be idempotent: the deduplication check is a separate concern handled at the infrastructure level. Combined with the transactional outbox pattern on the producer side, it enables exactly-once semantics end-to-end (at-least-once delivery + consumer-side deduplication = effectively exactly-once processing).

When to use

• Any consumer that processes non-idempotent operations — charging payments, sending notifications, decrementing inventory, publishing downstream events.
• Consumers of at-least-once brokers — SQS, Kafka (without transactional APIs), RabbitMQ.
• Saga participants — each step in a saga must be idempotent because the saga orchestrator may retry failed steps.
• Paired with transactional outbox — to achieve end-to-end exactly-once semantics across the producer and consumer.

When not to use

• When the consumer is already naturally idempotent — if processing the message twice has no additional effect (e.g., updating a field to the same value), the inbox overhead is not warranted.
• When the consumer has no relational database — the pattern relies on a transactional INSERT with a unique constraint. Redis SET NX can serve a similar purpose for stateless consumers but does not provide the same transactional guarantee.
• When Kafka transactional exactly-once is already configured — Kafka's transactional producer + consumer with read_committed isolation provides exactly-once within the Kafka ecosystem; the inbox pattern is an alternative for cases where that approach is not viable.

Typical architecture

The consumer receives a message. Before applying any side-effect, it attempts to insert the message ID into the inbox table within a database transaction. If the INSERT succeeds (no prior record), the consumer applies the business side-effect in the same transaction and commits. If the INSERT fails with a unique key violation, the message is a duplicate and the consumer ACKs it without applying the side-effect.

Consumer receives message:
  { "messageId": "msg-abc-123", "type": "payment.completed", "orderId": "ord-1" }

BEGIN TX
  -- Attempt deduplication check
  INSERT INTO inbox (message_id, processed_at)
  VALUES ('msg-abc-123', NOW())
  ON CONFLICT (message_id) DO NOTHING

  -- Check rows affected
  IF rows_affected == 0 THEN
    -- Duplicate detected, skip
    ROLLBACK
    ACK message to broker (discard safely)
    RETURN

  -- No conflict: first time processing
  UPDATE orders SET status = 'paid' WHERE id = 'ord-1'
  INSERT INTO outbox (...) VALUES (...)  -- optionally publish downstream event
COMMIT TX

ACK message to broker

-- Second delivery of same message:
BEGIN TX
  INSERT INTO inbox (message_id, ...) VALUES ('msg-abc-123', NOW())
  → UNIQUE CONSTRAINT VIOLATION → 0 rows affected
ROLLBACK
ACK (duplicate, safe to discard)

Pros and cons

Implementation notes

Inbox table schema: Minimal schema: message_id VARCHAR(255) PK (the unique broker message ID — SQS MessageId, Kafka offset+partition, or a business-level event ID), processed_at TIMESTAMP NOT NULL DEFAULT NOW(), optionally consumer_type VARCHAR if multiple consumer types share the same database. The primary key on message_id is the uniqueness constraint that drives the deduplication check.

Message IDs: Use the broker's native message ID if it is stable and unique (SQS MessageId, Kafka partition + offset string, RabbitMQ delivery tag is not suitable — use messageId property). For events published via the transactional outbox pattern, the outbox row's UUID is a stable, unique event ID that can serve as the inbox message ID. This ensures that even if the outbox relay re-delivers the same event, the inbox catches the duplicate.

Composite inbox key: If the same message ID could legitimately be processed by different consumers in the same database (e.g., multiple consumer types share a schema), use a composite key: (message_id, consumer_type).

Inbox TTL and pruning: The inbox table only needs to retain entries long enough to cover the maximum re-delivery window of the broker. For SQS, the maximum message retention period is 14 days — set the inbox TTL to at least 14 days. For Kafka, it depends on the topic retention configuration. Use a scheduled job to delete rows where processed_at < NOW() - INTERVAL '14 days'. Partition the table by processed_at month for fast partition drops.

Horizontal scaling with competing consumers: Multiple consumer instances can safely use the same inbox table concurrently. If two instances receive the same message simultaneously (race condition), only one INSERT will succeed; the other will get a unique constraint violation and safely skip the message. The database uniqueness constraint provides the serialisation without any consumer-level coordination.

Combining with the transactional outbox (exactly-once end-to-end): The producer writes to the outbox atomically with the business write. The outbox relay publishes at-least-once to the broker. The consumer uses the inbox to deduplicate at the consumer side. The result: even if the outbox relay re-delivers the same event multiple times, the consumer will process the business logic exactly once. This is the closest practical approximation to exactly-once semantics in a distributed system without Kafka transactions.

Common failure modes

• ACKing the message before the transaction commits: If the consumer ACKs the message to the broker before the DB transaction commits and the DB transaction subsequently fails, the message is lost — neither processed nor redelivered. Always ACK after a successful commit.
• Inbox table never pruned: Over time the table grows to millions of rows. INSERT performance degrades as the index grows. Implement pruning or range partitioning from day one.
• Message ID not stable or not unique: Using a non-stable ID (e.g., RabbitMQ delivery tag which changes on re-delivery, or a wall-clock timestamp) breaks the deduplication check. Use a stable, immutable message ID generated at publish time.
• Deduplication window too short: A message is re-delivered after the inbox TTL; the inbox row has been pruned; the message is treated as new and double-processed. Set the TTL to exceed the broker's maximum retention period.

Decision checklist

• Does the inbox INSERT and the business side-effect execute in the same database transaction?
• Is the message ACKed to the broker only after the database transaction successfully commits?
• Is a stable, immutable message ID used (not the RabbitMQ delivery tag or a timestamp)?
• Is the inbox TTL set to at least the maximum broker message retention period?
• Is there a scheduled pruning job (or range partition drop) to prevent unbounded inbox table growth?
• Is the inbox table tested under concurrent consumer load to verify the uniqueness constraint race condition is handled correctly?

Example use cases

• Payment charging: Payment consumer receives payment.requested; uses inbox to ensure the charge is attempted exactly once even if the message is re-delivered after a consumer crash mid-charge.
• Email notification service: Welcome email consumer deduplicates user.registered events; even if the event is re-delivered multiple times, the user receives exactly one welcome email.
• Inventory decrement: Fulfilment consumer decrements inventory for order.confirmed; inbox prevents double-decrement when the outbox relay re-delivers the event after a relay crash.

Related patterns

• Transactional Outbox Pattern — the producer-side complement; together they provide end-to-end exactly-once semantics.
• Competing Consumers — multiple competing consumers safely share the same inbox table; the uniqueness constraint serialises duplicate attempts.
• Message Queues — the at-least-once broker that makes the inbox pattern necessary.

Further reading

• Idempotent Consumer Pattern — Chris Richardson's description on microservices.io.
• The Outbox Pattern (Kamil Grzybek) — detailed walkthrough of both outbox and inbox patterns with code examples.
• SQS Message Attributes and MessageId — SQS native message ID suitable for use as inbox key.