Foundations Essential

Idempotency

An operation that produces the same result whether applied once or many times — the essential property for safe retries in distributed systems.

⏱ 7 min read

What it is

An operation is idempotent if applying it multiple times produces the same result as applying it once. Formally, for a function f, idempotency means f(f(x)) = f(x). In the context of distributed systems and APIs, an operation is idempotent if calling it once and calling it ten times with the same input leaves the system in the same final state. Setting a value is idempotent (SET x = 5 twice results in x = 5). Incrementing is not (INCREMENT x twice results in x = initial + 2).

Idempotency is essential in distributed systems because network communication is inherently unreliable. When a client sends a request and does not receive a response (due to timeout, network interruption, or server crash), it cannot know whether the operation was executed. If the operation is not idempotent, retrying it could cause duplicate effects: charging a customer twice, creating duplicate records, or sending the same email twice. Making operations idempotent means retries are always safe — the worst outcome is redundant work, not data corruption.

Why it exists

Message delivery in distributed systems can be "at most once" (risk of loss), "at least once" (risk of duplicates), or "exactly once" (expensive, complex, and often illusory). In practice, the most reliable and scalable systems use at-least-once delivery — they guarantee every message is processed but may deliver it more than once due to retries after failures. Idempotent consumers make at-least-once delivery semantically equivalent to exactly-once: a message processed twice has the same effect as processing it once.

HTTP, the foundational protocol of the web, explicitly designates which methods should be idempotent (GET, HEAD, PUT, DELETE) and which are not (POST). This design makes web browsers and HTTP clients safe to retry failed requests for idempotent methods, and gives developers a clear signal that POST requests require special care to avoid duplicate side effects.

When to use

All payment and financial operations — charge attempts, refund requests, and fund transfers must be idempotent to prevent duplicate charges or transfers.
Any API endpoint that triggers a side effect (send email, create record, trigger workflow) and may be called by retry-capable clients.
Message queue consumers that process at-least-once delivered messages.
Webhook receivers — webhooks are commonly retried by the sender on non-2xx responses, making the receiver responsible for deduplication.
Distributed workflow steps — any step in a saga or choreography-based transaction that might be retried after a failure.

When not to use

Idempotency is never "not applicable" — if an operation has side effects, it should always be designed for idempotency unless duplicates are explicitly acceptable and rare.
Pure reads (GET, SELECT) are naturally idempotent — no additional mechanism is required.
If "exactly once" processing is truly required (not "at-least-once with idempotent consumers"), Kafka exactly-once semantics or transactional outbox patterns are the appropriate tools — not just idempotency keys.

Typical architecture

The most common implementation pattern for idempotent APIs is the idempotency key: the client generates a unique identifier (typically a UUID) for each logical operation and includes it in the request. The server uses this key to detect and short-circuit duplicate requests. The sequence is: check key store → if key exists, return cached response; if key does not exist, execute operation, store key with response, return response.


Client                           Server
  │                                │
  │  POST /payments                │
  │  Idempotency-Key: uuid-abc123  │
  │  { amount: 99.99 }             │
  │ ─────────────────────────────► │
  │                                ├─► Check key store
  │                                │     ┌────────────────────────┐
  │                                │     │ idempotency_keys table │
  │                                │     │ key | response | ttl   │
  │                                │     └────────────────────────┘
  │                                │
  │  [First call: key not found]   │
  │                                ├─► Execute payment
  │                                ├─► Store key + response in DB
  │                                │   (atomic with payment commit,
  │  ◄─────────────────────────── │    or in same transaction)
  │  201 { payment_id: "pay_456" } │
  │                                │
  │  [Network timeout — retry]     │
  │                                │
  │  POST /payments                │
  │  Idempotency-Key: uuid-abc123  │
  │  { amount: 99.99 }             │
  │ ─────────────────────────────► │
  │                                ├─► Check key store
  │                                │     Key found! Return cached response.
  │  ◄─────────────────────────── │     Payment NOT executed again.
  │  201 { payment_id: "pay_456" } │

HTTP Method idempotency (by spec):
  GET    → Idempotent (no side effects)
  HEAD   → Idempotent (same as GET, no body)
  PUT    → Idempotent (sets resource to exact state)
  DELETE → Idempotent (second delete of same resource = 404, same state)
  POST   → NOT idempotent (use Idempotency-Key header for safety)
  PATCH  → NOT idempotent by spec (but can be made so by design)

Pros and cons

Pros

Safe retries — clients can retry failed or timed-out requests without risk of duplicate side effects, dramatically simplifying error handling.
Enables at-least-once delivery — the simplest and most scalable message delivery guarantee becomes safe to use.
Simplifies distributed transactions — retry-safe operations remove the need for complex two-phase commit protocols in many scenarios.
Improves resilience — transient failures (network blips, pod restarts) can be recovered automatically with retries.

Cons

Storage overhead — idempotency key storage requires a persistent, fast key-value store or database table with appropriate TTLs and cleanup.
Idempotency window — keys are typically stored for a finite TTL (24 hours is common for payments). Retries after the window may re-execute the operation.
Side effects that are inherently non-idempotent — sending an email or triggering a physical process cannot be "undone" by deduplication after the fact.
Distributed key storage — in a clustered server environment, all servers must share the same idempotency key store to prevent one server from executing what another has already completed.

Implementation notes

Idempotency key storage: The idempotency key table must be stored in a database that is visible to all instances of the service. Using a local in-memory store defeats the purpose in a multi-instance deployment. The key and its associated cached response should be committed atomically with the operation itself — if you charge the card in the same database transaction in which you insert the idempotency key, you can never have a state where the charge succeeded but the key was not stored (or vice versa). For operations that span external systems (e.g., charging a payment provider), use the outbox pattern to ensure the idempotency record is committed before the external call is confirmed.

Database-level idempotency: SQL databases support idempotent write patterns natively. INSERT INTO orders (...) ON CONFLICT (idempotency_key) DO NOTHING (PostgreSQL) atomically either creates the record or does nothing if the key already exists. INSERT OR IGNORE in SQLite, and INSERT IGNORE in MySQL serve the same purpose. For updates, UPDATE orders SET status='paid' WHERE order_id=? AND status='pending' is idempotent because the WHERE clause means the second update matches no rows if the first already changed the status. Optimistic locking via version columns is another approach: UPDATE orders SET status='paid', version=version+1 WHERE order_id=? AND version=?.

Idempotent message consumers: In event-driven systems with at-least-once delivery (Kafka, SQS, RabbitMQ), consumers must handle duplicate messages. The standard pattern is a deduplication table keyed on the message ID: before processing, check if the message ID exists in the table; if yes, acknowledge and skip; if no, process and insert the ID in the same transaction. Kafka's log compaction and consumer offset tracking provide exactly-once within a consumer group, but the application-level consumer still needs idempotency for external side effects like database writes or HTTP calls.

Common failure modes

Non-idempotent side effects: An idempotency key deduplicates the API call, but the implementation still sends an email confirmation for each call. The email send must also be guarded — typically by checking a "notification_sent" flag before sending.
Idempotency key not tied to operation: Using a client-generated key for the HTTP call but not linking it to the specific operation (e.g., allowing the same key to be used for different amounts). The server must validate that a duplicate key has identical request parameters, not just the same key.
Idempotency window too short: A 1-hour TTL on idempotency keys means a payment retry 2 hours later (e.g., from a mobile app that was offline) will re-execute. Stripe uses a 24-hour window; for financial operations, longer windows are safer.
Non-shared key store in distributed deployments: Each pod maintains its own in-memory idempotency store. Load balancer routes the retry to a different pod that has no record of the first request — the operation executes twice.

Decision checklist

Does this operation have side effects that should not be repeated on retry (charge, create, send, trigger)?
Can the client include an idempotency key with the request — either auto-generated or provided as a request parameter?
Is the idempotency key store shared across all service instances (not per-pod in-memory)?
Is the idempotency key committed atomically with the operation to prevent key/operation split-brain?
Is the idempotency TTL appropriate for the retry window of the clients that call this service?
Are all downstream side effects (emails, external API calls, event publications) also guarded against duplication?
For message consumers: is the message ID used as a deduplication key in a persistent store, not just checked in memory?

Example use cases

Stripe's payment API requires clients to include an Idempotency-Key header with every charge request. If a request times out, the client retries with the same key and Stripe returns the cached result of the first (and only) charge.
An order placement service uses INSERT INTO orders ... ON CONFLICT (idempotency_key) DO NOTHING to ensure that a mobile app that retries a timed-out order submission does not create duplicate orders.
An SQS consumer processing webhook events stores the SQS message ID in a DynamoDB deduplication table. If SQS delivers the same message twice (a guaranteed possibility with standard queues), the second delivery is detected and skipped without re-processing.
A Kubernetes controller uses idempotent reconciliation — the reconcile loop runs continuously and must be safe to run multiple times. Each reconcile call checks the desired state vs the current state and only takes action if a delta exists, making repeated runs safe.

Eventual Consistency — at-least-once delivery in eventually consistent systems is the primary driver for needing idempotent consumers.
Backward Compatibility — idempotency keys and retry behavior are part of the API contract and must be considered during API evolution.