Reliability & Resilience Advanced

Exactly-Once Delivery

Guaranteeing that a message is processed exactly one time — neither lost nor processed as a duplicate.

⏱ 11 min read

What it is

Exactly-once delivery is the strongest message delivery guarantee: a producer's message is delivered to a consumer and its effects are applied to the system state precisely one time, with no possibility of loss (at-least-once risk) or duplication (at-most-once risk). It is a compound guarantee comprising two properties: exactly-once delivery (the broker delivers the message exactly once end-to-end) and exactly-once processing (the consumer's side effects are applied exactly once even if the message is delivered or processed multiple times due to failures).

True exactly-once delivery across arbitrary distributed systems is impossible under all failure models (the "exactly-once myth"), but practical exactly-once semantics can be achieved through a combination of idempotent producers, transactional consumers, and distributed coordination — with the caveat that this applies within specific component boundaries. Kafka's transactional API provides exactly-once semantics within a Kafka-to-Kafka read-process-write loop. AWS SQS FIFO with deduplication and idempotent consumers achieves effectively-exactly-once processing. True cross-system exactly-once (e.g., database + message broker atomicity) requires the Transactional Outbox pattern or two-phase commit.

Why it exists

Many business operations are non-idempotent by nature: charging a payment card twice causes real financial harm; updating an account balance twice corrupts data; sending a confirmation email twice confuses customers. At-least-once delivery — the default for most reliable messaging systems — accepts the possibility of duplicates and shifts the burden of deduplication to the consumer via idempotency. But building idempotent consumers for every operation is complex and error-prone. Exactly-once delivery aims to eliminate that burden at the infrastructure level, simplifying application logic for scenarios where duplicate processing is genuinely unacceptable.

When to use

Financial transactions where duplicate processing causes direct monetary harm (double charges, double credits).
Stateful stream processing with Kafka where read-process-write loops must not produce duplicate results (aggregations, joins, windowed computations).
Inventory management where a duplicate decrement results in incorrect stock counts that affect physical fulfilment.
Any bounded exactly-once scope where the infrastructure cost (transactional producers, Kafka transactions) is justified by the criticality of the operation.

When not to use

Operations that are already naturally idempotent (setting a value to a specific state) — at-least-once + idempotent processing is simpler and cheaper.
High-throughput, low-value streams (analytics events, metrics, logs) where the performance cost of transactions is not justified.
Cross-system exactly-once (message broker + relational DB) without a Transactional Outbox — this cannot be achieved reliably without distributed 2PC.
Systems using brokers that do not support transactional or deduplication semantics natively — without broker support, the complexity of achieving exactly-once falls entirely on the application.

Typical architecture

Kafka Exactly-Once Semantics (EOS) — Read-Process-Write
═══════════════════════════════════════════════════════════

Producer (transactional)            Kafka Broker
────────────────────────            ──────────────────────────────
initTransactions()
beginTransaction()
  read from source topic  ──────►   Source Topic (input events)
  process event
  produce to sink topic   ──────►   Sink Topic (results)
  commitSync offset       ──────►   Consumer Group Offsets (atomic)
commitTransaction()
                                    All three writes are atomic:
                                    - sink message published
                                    - offset committed
                                    → consumer never re-processes
                                      or loses a processed event

SQS FIFO with Deduplication
────────────────────────────────────
Producer → SendMessage with MessageDeduplicationId=hash(content)
           │
           ▼
SQS deduplication window (5 minutes)
  - Duplicate MessageDeduplicationId suppressed by broker
  - Consumer receives each logical message exactly once
  - Consumer must still be idempotent for cross-window replays

Pros and cons

Pros

Eliminates duplicate processing without requiring each consumer to implement deduplication logic individually.
Simplifies application code for operations where duplicate application would cause correctness or financial errors.
Kafka's EOS provides auditable, transaction-scoped processing that can be replayed to a consistent point.
SQS FIFO deduplication integrates transparently at the broker level — no consumer code change required for the deduplication window.

Cons

Significant performance overhead: Kafka EOS adds latency due to transactional coordination; SQS FIFO has lower throughput than standard queues.
Exactly-once semantics only apply within bounded components — a Kafka EOS producer cannot atomically commit to a database without Transactional Outbox.
Complex failure recovery: transactional aborts must roll back all writes atomically; partial failures leave the system in inconsistent state if not handled correctly.
SQS FIFO deduplication is scoped to a 5-minute window — duplicates outside this window still require idempotent consumers.
"Exactly-once" is a marketing term that must be examined carefully per-broker — the scope and guarantees vary significantly across implementations.

Implementation notes

For Kafka exactly-once semantics, enable enable.idempotence=true on the producer (required) and set transactional.id to a stable, unique identifier per producer instance. Set isolation.level=read_committed on consumers to ensure they never read messages from aborted transactions. Note that EOS adds approximately 15–30% latency overhead due to transactional coordination — benchmark carefully before adopting in high-throughput paths. Kafka Streams and Apache Flink provide higher-level EOS abstractions that handle transactional producer lifecycle automatically.

For SQS FIFO, set MessageDeduplicationId to a deterministic hash of the message content or a business key (order ID, payment reference). The deduplication window is 5 minutes — within this window, any message with the same deduplication ID is silently dropped by the broker. Beyond 5 minutes, the consumer must implement idempotency using a persistent deduplication store keyed on the same business ID. ContentBasedDeduplication (enabled at queue level) computes the hash automatically from message body, but requires all logically identical messages to have identical bodies.

Common failure modes

Scope confusion: believing Kafka EOS guarantees atomicity with external databases — it only guarantees atomicity within the Kafka topic-to-topic scope.
Transactional ID reuse: multiple producer instances sharing the same transactional.id causes epoch conflicts and broker fencing of one producer.
read_committed omitted: consumers using read_uncommitted (the default) read messages from aborted transactions, causing exactly-not-once processing.
SQS FIFO throughput limits: FIFO queues have a 3,000 messages/second limit (with batching) versus standard queues' effectively unlimited throughput — using FIFO for high-throughput paths causes throttling.
Relying on 5-minute window: treating the SQS FIFO deduplication window as a permanent guarantee, and not implementing idempotent consumers for the broader case.

Decision checklist

Is the operation genuinely non-idempotent and would duplicates cause correctness errors or financial harm?
Is the scope of exactly-once bounded to a component pair that supports it (Kafka-to-Kafka, SQS FIFO-to-consumer)?
Have you benchmarked the performance overhead of transactional production and confirmed it is acceptable?
For Kafka EOS: is isolation.level=read_committed set on all consumers of the output topic?
For SQS FIFO: is the deduplication ID deterministic and collision-resistant for the message business key?
For operations spanning message broker + database: is the Transactional Outbox pattern being used instead?
Is the exactly-once boundary documented so future developers understand its scope and limitations?

Example use cases

A Kafka Streams payment aggregation job uses EOS to ensure that each payment event is counted exactly once in the running balance, even when the job restarts after a crash mid-batch.
An SQS FIFO queue with content-based deduplication routes order creation events to a fulfilment service, ensuring that network retries from the producer do not create duplicate orders within the 5-minute window.
Apache Flink's checkpointing mechanism combined with Kafka transactional sinks provides exactly-once output for a fraud detection pipeline that writes alerts to a results topic.

At-Least-Once Delivery — the more common guarantee; exactly-once is built on top of at-least-once with deduplication.
Idempotency — when exactly-once at the broker level is impossible, idempotency at the consumer level achieves the same effect.
Transactional Outbox — achieves effectively-exactly-once for database + message broker spans.