Reliability & Resilience

At-Least-Once Delivery

The messaging guarantee that every message will be delivered one or more times, accepting duplicates in exchange for no message loss.

⏱ 9 min read

What it is

At-least-once delivery is a messaging guarantee in which the broker commits to delivering a message to the consumer at least one time — but possibly more than once. If the broker does not receive a positive acknowledgement (ACK) from the consumer within a defined visibility timeout, it assumes the consumer failed and redelivers the message to another consumer or the same consumer on retry. This redelivery can happen multiple times in the presence of failures: network partitions, consumer crashes, slow processing that exceeds visibility timeout, or abnormal shutdowns before the consumer sends its ACK.

At-least-once is the default delivery mode for most production-grade message brokers: SQS (standard), RabbitMQ with manual ACK, and Kafka with consumer offset management all implement at-least-once by default. At-at-most-once delivery (fire-and-forget) is simpler but loses messages on failure. Exactly-once delivery eliminates duplicates but is costly and bounded. At-least-once is the pragmatic middle ground: no message loss, duplicates possible, consumer must be idempotent. Most production message-driven architectures operate on at-least-once delivery with idempotent consumers.

Why it exists

In a distributed system with unreliable networks and fallible processes, there is no way for a broker to know with certainty whether a consumer received, processed, and committed a message successfully without an acknowledgement. Network packets can be lost in either direction (delivery or ACK). Consumer processes can crash after processing but before sending ACK. To guarantee that no message is lost, the broker must retry delivery until it receives an ACK — which necessarily means it may redeliver messages that were actually processed successfully but whose ACK was lost. At-least-once delivery accepts this tradeoff: it eliminates message loss at the cost of occasional duplicates.

When to use

Any system where message loss is unacceptable (orders, payments, notifications, state changes) — the default choice for reliable messaging.
Systems where consumer processing is or can be made idempotent — at-least-once + idempotent consumer is the most common pattern in production systems.
Kafka-based pipelines where consumer offsets provide the at-least-once guarantee and Transactional Outbox or idempotency keys handle duplicate side effects.
Event-driven architectures where events drive downstream state updates and each event carries a stable event ID for deduplication.

When not to use

When consumer operations are inherently non-idempotent and cannot be made idempotent — use exactly-once semantics instead.
High-frequency telemetry, metrics, or analytics events where occasional loss is acceptable and the overhead of ACK is not worthwhile — use at-most-once.
When the consumer system cannot tolerate any duplicate data, even transient duplicates visible before deduplication catches them — requires exactly-once or synchronous deduplication at the consumer.

Typical architecture

At-Least-Once Delivery with Idempotent Consumer
════════════════════════════════════════════════════

Broker               Consumer                  Database
──────               ────────                  ────────
Deliver msg ───────► Receive msg
                     Process msg
                     Apply side effect ──────► Write with idempotency key
                                               INSERT ... ON CONFLICT IGNORE
                     Send ACK ◄───────────────── confirm write
◄──────── ACK ───────
Delete from queue

Failure scenario: Consumer crashes after write, before ACK
──────────────────────────────────────────────────────────
Broker             Consumer                  Database
──────             ────────                  ────────
Deliver msg ──────► crash                   side effect applied
visibility timeout expires
Redeliver msg ────► Receive duplicate
                    Apply side effect ──────► INSERT ... ON CONFLICT IGNORE
                                              (duplicate ignored by DB)
                    Send ACK ◄────────────────── confirm
◄───── ACK ──────────
Delete from queue       ✓ Exactly-once effect, at-least-once delivery

Pros and cons

Pros

Guarantees no message loss — every message is eventually processed as long as consumers are available.
Default and well-supported by all major message brokers — no special configuration required.
Simple to reason about: the broker handles reliability; the consumer handles idempotency.
No performance overhead of distributed transactions required by exactly-once semantics.
Well-understood failure modes — duplicate processing is predictable and testable.

Cons

Requires all consumers to implement idempotency — a correctness requirement that must be maintained as the system evolves.
Duplicate messages under normal operation are rare but not zero — systems must be tested for correct duplicate handling under failure conditions.
Visibility timeout tuning is critical: too short causes excessive redeliveries for slow consumers; too long means failed consumer takes longer to recover.
Deduplication stores (used to track processed message IDs) must be highly available and fast, or they become a bottleneck.

Implementation notes

For SQS, set the visibility timeout to 6× the expected maximum processing time for a single message. If processing takes up to 10 seconds, set visibility timeout to 60 seconds. Use ChangeMessageVisibility to extend the timeout for messages that require longer processing. Set message retention period long enough that a temporary consumer outage (deployment, incident) does not cause messages to expire before processing resumes — 4 days is the default; 14 days is the maximum.

Implement idempotency in the consumer using a deduplication key stored in a Redis set, DynamoDB table, or relational DB unique constraint. For SQS standard queues, use the MessageId (UUID assigned by SQS) as the deduplication key. For SQS FIFO, use the MessageDeduplicationId. For Kafka, use a combination of topic name + partition + offset as a stable, globally unique message ID. Structure the deduplication check + business logic + deduplication mark as an atomic unit: check-then-act requires care to avoid race conditions in concurrent consumer environments.

Common failure modes

Visibility timeout too short: slow consumers time out repeatedly, causing the message to be redelivered to other consumers concurrently, leading to parallel duplicate processing that idempotency must handle correctly.
Non-idempotent consumer: the consumer applies side effects without a deduplication check, causing duplicates (double charges, duplicate inserts, duplicate notifications) when messages are redelivered.
Deduplication store unavailable: if the Redis deduplication store is down, the consumer either fails all processing (availability impact) or processes without deduplication (correctness risk).
ACK lost but processing complete: consumer processed the message and committed results, but the ACK was lost (network issue), causing the broker to redeliver. Without idempotency, this causes a duplicate.
Long processing with no heartbeat: consumer processes a complex message for 20 minutes but visibility timeout is 30 seconds — the broker redelivers to another consumer at 31 seconds while processing is still ongoing.

Decision checklist

Is every consumer operation idempotent — safe to execute multiple times with the same result?
Is visibility timeout set to at least 6× the expected maximum per-message processing time?
Is there a deduplication mechanism (idempotency key + DB constraint or deduplication store) in place?
Are duplicate scenarios tested in integration tests — not just happy path?
Is the deduplication store highly available and fast enough not to be a bottleneck?
Is there a DLQ configured for messages that exceed maxReceiveCount after genuine failures?
Is message retention set long enough to survive planned downtime (deployments, incidents)?

Example use cases

An SQS-based order processing consumer: each order message has a unique order ID; the consumer performs INSERT INTO orders (...) ON CONFLICT (order_id) DO NOTHING to ensure the same order is never created twice even under redelivery.
A Kafka consumer for user analytics events: each event has a UUID; the consumer checks a Redis bloom filter for seen event IDs before processing, skipping duplicates efficiently at scale.
A notification consumer that sends emails: the consumer checks a notifications_sent table keyed by (user_id, notification_id) before sending; idempotent at the application layer prevents duplicate emails on redelivery.

Exactly-Once Delivery — the stronger guarantee that eliminates duplicates at the infrastructure level.
Idempotency — the consumer-side pattern that makes at-least-once delivery safe.
Dead Letter Queues — where messages go after exhausting at-least-once retries.