Transactional Outbox Pattern

What it is

The transactional outbox pattern solves one of the fundamental reliability problems in event-driven microservices: a service must atomically update its database and publish an event to the message broker, but these are two separate systems and cannot participate in the same ACID transaction. The outbox pattern avoids this by writing the event to an outbox table in the same database transaction as the business data. A separate relay process reads unpublished rows from the outbox table and publishes them to the broker, then marks them as published. The relay may use polling or Change Data Capture (CDC) via Debezium to detect new outbox rows.

The result is that the database write and the event publication become eventually consistent with at-least-once delivery guaranteed: if the service commits the DB transaction, the event will eventually be published. If the service crashes before commit, neither the DB change nor the event is visible — the atomicity guarantee of the local database transaction eliminates the race condition.

Why it exists

The dual-write problem is insidious: a service writes to the database, then publishes a message to Kafka. Between these two operations, the service can crash, the broker can be unavailable, or the network can fail. The result is a database that has the updated state but no message published (silent data loss from the broker's perspective) or, if the order is reversed, a message published for a database write that was rolled back (phantom events). Both corrupt downstream systems. The transactional outbox pattern is the standard solution for this class of problem in microservices — it appears in virtually every event-driven architecture guide and is directly supported by frameworks like Eventuate Tram, Axon, and Debezium's outbox router.

When to use

• Any service that must update a database and publish an event atomically — the canonical use case: order service must commit the order and publish order.created.
• Financial systems where silent event loss is unacceptable — payment completed, inventory reserved, balance updated — all must result in reliably published events.
• Saga orchestration — the outbox ensures each saga step's database update and subsequent command/event publication are atomic.
• When you cannot use 2PC or distributed transactions — this is the practical alternative to XA transactions in microservices.

When not to use

• When eventual consistency is unacceptable — the outbox relay introduces a small lag (typically milliseconds to seconds) between the DB commit and event publication. For hard real-time requirements, this may be insufficient.
• When you have no relational database — the pattern relies on local ACID transactions. If your primary store is a NoSQL database without multi-document transactions, the pattern does not directly apply (though document stores with transaction support, like MongoDB 4.x+, can use a variant).
• Very high-throughput services with no tolerance for outbox table growth — the outbox table must be pruned; at very high insert rates, this requires careful partitioning.

Typical architecture

The service writes to the business table and the outbox table in a single transaction. The relay process queries the outbox for unpublished rows (polling) or reads the DB change log via CDC (Debezium). The relay publishes each event to the broker, then marks it as published in the outbox. Processed rows are pruned on a schedule.

Service (within single DB transaction):
  BEGIN TX
    INSERT INTO orders (id, status, ...) VALUES ('ord-1', 'confirmed', ...)
    INSERT INTO outbox (id, event_type, aggregate_id, payload, published)
           VALUES (uuid(), 'order.confirmed', 'ord-1',
                  '{"orderId":"ord-1","total":99.99}', false)
  COMMIT TX

Relay Process (polling variant):
  LOOP every 500ms:
    SELECT * FROM outbox WHERE published = false ORDER BY created_at LIMIT 100
    FOR EACH row:
      broker.publish(topic=row.event_type, key=row.aggregate_id, payload=row.payload)
      UPDATE outbox SET published=true, published_at=NOW() WHERE id=row.id

Relay Process (CDC variant - Debezium):
  Debezium reads PostgreSQL WAL → detects INSERT on outbox table
  → publishes directly to Kafka topic via Debezium Outbox Router
  No polling, sub-second latency, no extra DB queries

Pros and cons

Implementation notes

Outbox table schema: A minimal outbox table: id UUID PK, event_type VARCHAR (e.g., order.confirmed), aggregate_type VARCHAR (e.g., Order), aggregate_id VARCHAR (e.g., the order ID — also used as the Kafka partition key to preserve per-aggregate ordering), payload JSONB (the full event payload), published BOOLEAN DEFAULT false, created_at TIMESTAMP, published_at TIMESTAMP NULL. Add an index on (published, created_at) for the polling query.

Polling relay vs CDC-based relay: Polling is simpler to implement (a cron job or background thread) but polls the database repeatedly, adding load. Set the poll interval to at least 500 ms to avoid overwhelming the DB connection pool. CDC-based relay using Debezium reads the database transaction log (WAL for PostgreSQL, binlog for MySQL) and emits change events in near-real-time (<100ms latency) without querying the database. Debezium's Outbox Event Router SMT (Single Message Transform) can transform the outbox row into a properly keyed Kafka message automatically.

Pruning the outbox table: Never let the outbox table grow indefinitely. Use a scheduled job to delete rows where published = true AND published_at < NOW() - INTERVAL '7 days'. Alternatively, use range partitioning on created_at and drop old partitions wholesale (faster than deleting individual rows).

Ordering guarantee: Use the aggregate_id as the Kafka partition key to ensure all events for the same aggregate (e.g., same order ID) are published to the same Kafka partition, preserving temporal ordering for that aggregate.

At-least-once and consumer idempotency: The relay provides at-least-once delivery, not exactly-once. If the relay crashes after publishing but before updating published = true, it will re-publish the same event on restart. All consumers downstream must be idempotent. Pair with the inbox pattern for exactly-once semantics end-to-end.

Common failure modes

• Relay not deployed / crashes silently: Events accumulate in the outbox table indefinitely. Alert on outbox table row count exceeding a threshold (e.g., >1,000 unpublished rows) as a signal that the relay is not running.
• Outbox table never pruned: The table grows to millions of rows, slowing insert performance and polling queries. Implement pruning from day one.
• Non-idempotent consumers receiving re-delivered events: The relay re-delivers after a crash; consumers without idempotency guards double-process events. Implement the inbox pattern on all critical consumers.
• Aggregate ordering violated: Relay publishes events without using the aggregate ID as the partition key; unrelated events for the same aggregate land on different Kafka partitions, violating per-aggregate ordering.

Decision checklist

• Is the outbox INSERT in the same database transaction as the business data update?
• Is the relay using CDC (Debezium) or a polling relay with appropriate poll interval?
• Is the aggregate ID used as the Kafka partition key to preserve per-aggregate ordering?
• Is there a scheduled pruning job to delete old published outbox rows?
• Is there an alert on the count of unpublished outbox rows to detect relay failures?
• Are all downstream consumers idempotent (safe for at-least-once delivery)?
• Is the inbox pattern applied at critical consumers for exactly-once end-to-end semantics?

Example use cases

• Order service: Commits order to orders table and inserts order.confirmed event to outbox table atomically; Debezium CDC relay publishes to Kafka; inventory service consumes and reserves stock.
• Payment service: Writes payment record and inserts payment.completed to outbox in one transaction; relay publishes to broker; fulfilment service proceeds with shipment only after consuming the reliable payment event.
• User registration: Inserts user record and user.registered outbox event atomically; relay publishes; email service sends welcome email; analytics service records new user — all from one reliable event.

Related patterns

• Inbox Pattern — the consumer-side complement: deduplicate re-delivered outbox events for exactly-once end-to-end semantics.
• Message Queues — the broker the outbox relay publishes to.
• Correlation IDs — embed correlation IDs in the outbox payload for end-to-end traceability.

Further reading

• Transactional Outbox Pattern — Chris Richardson's canonical definition on microservices.io.
• Debezium Outbox Event Router — CDC-based outbox relay implementation.
• Eventuate Tram — framework that implements the transactional outbox pattern for Java microservices.