Reliability & Resilience Essential

Retry Pattern

Reattempting failed operations when the cause is transient and the operation is safe to repeat.

⏱ 10 min read

What it is

The Retry pattern automatically re-executes a failed operation after a brief wait, under the assumption that the failure is transient — caused by a momentary network hiccup, a short-lived service overload, or a rate-limit response — and will likely resolve itself without structural intervention. The pattern wraps the call site with retry logic that tracks attempt count, evaluates whether the failure is retryable, and applies a delay strategy between attempts. Most production retry implementations use exponential backoff with jitter (see the next article) rather than a fixed wait interval, to prevent all retrying callers from hitting the downstream service simultaneously.

Retry is distinct from polling or idempotent replay: it applies at the call site immediately after a failure, within a single logical operation, rather than being driven by a background scheduler. The pattern is ubiquitous in HTTP clients, gRPC stubs, database drivers, and message broker SDKs — but when applied naively without caps or selectivity, it can amplify load on already struggling systems rather than relieve them.

Why it exists

Distributed systems experience transient failures constantly: a load balancer briefly drops a connection during a rolling restart, a DNS resolver times out under burst load, a managed database instance fails over and is unreachable for three seconds, or an upstream API returns HTTP 503 because it is momentarily throttled. Without automatic retries, every such brief disruption becomes a visible error for the calling client. The Retry pattern absorbs these brief failures transparently, dramatically improving perceived reliability without any structural change to the downstream system.

The pattern also aligns with the design philosophy of autonomous services: a caller should not require manual operator intervention every time a downstream dependency experiences a brief blip. Combined with exponential backoff and circuit breakers, retries form the first tier of resilience defence for inter-service communication.

When to use

  • The operation is idempotent — executing it multiple times produces the same result as executing it once (GET requests, PUT with full resource, Kafka producer with idempotent flag).
  • The error is transient — HTTP 429 Too Many Requests, HTTP 503 Service Unavailable, connection timeouts, DNS resolution failures, or database connection pool exhaustion that clears quickly.
  • The downstream service is expected to recover on its own without external action (e.g., rolling deployment completing, autoscaling adding capacity).
  • The operation has a defined maximum latency budget that can accommodate a small number of retries without exceeding the overall SLA.
  • You are calling external APIs (cloud SDKs, payment gateways) that explicitly document retry-safe endpoints.

When not to use

  • The operation is not idempotent — retrying a payment charge, a non-idempotent POST, or a state machine transition could cause duplicate side effects.
  • The failure is permanent — HTTP 400 Bad Request (malformed input), HTTP 401 Unauthorized, HTTP 403 Forbidden, HTTP 404 Not Found, or HTTP 422 Unprocessable Entity will not resolve with retries and should surface immediately.
  • The downstream is in a prolonged outage — unbounded retries under sustained failure amplify load; a circuit breaker should open instead.
  • The retry budget has been exhausted — continuing to retry beyond the cap wastes resources and increases tail latency for the caller.
  • The SLA does not have headroom — if a 200 ms p99 budget leaves no room for even one retry, fail fast instead.

Typical architecture

Retry Flow with Exponential Backoff
═════════════════════════════════════════════════════

Client                   Retry Wrapper              Downstream
  │                           │                          │
  │──── call() ──────────────►│                          │
  │                           │──── attempt 1 ──────────►│
  │                           │◄─── 503 ─────────────────│
  │                           │                          │
  │                           │  [transient? YES]        │
  │                           │  [attempt ≤ maxRetries?] │
  │                           │  wait: base * 2^0 + jitter│
  │                           │                          │
  │                           │──── attempt 2 ──────────►│
  │                           │◄─── 503 ─────────────────│
  │                           │  wait: base * 2^1 + jitter│
  │                           │                          │
  │                           │──── attempt 3 ──────────►│
  │                           │◄─── 200 OK ──────────────│
  │◄── success ───────────────│                          │

Error Classification
────────────────────
  Retry    → 429, 503, 504, connection timeout, DNS error
  No Retry → 400, 401, 403, 404, 422, business logic errors

Pros and cons

Pros

  • Transparently absorbs transient failures without surfacing errors to users or callers.
  • Simple to implement and available in most HTTP and gRPC client libraries out of the box.
  • Dramatically improves availability metrics when downstream services have brief restarts or brief throttle windows.
  • Decouples the caller from having to implement manual retry logic at the business layer.
  • Composable with circuit breakers and fallbacks for layered resilience.

Cons

  • Amplifies load on an already struggling downstream when not capped by a retry budget or circuit breaker.
  • Increases tail latency — a caller waiting through three retries with backoff may experience 10× the normal p99 latency.
  • Dangerous for non-idempotent operations — duplicate charges, duplicate inserts, duplicate emails are common retry bugs.
  • Retry storms occur when many callers simultaneously retry against the same service — jitter and circuit breakers must both be used.
  • Masks root causes — metrics showing high retry rates may obscure underlying infrastructure problems that need fixing.

Implementation notes

Always classify errors before retrying. Maintain two distinct sets: retryable errors (transient) and non-retryable errors (permanent). Retrying on a 400 Bad Request wastes resources and delays surfacing a bug; retrying on a 401 can lock accounts. In HTTP, 429 and 503 responses should also be checked for a Retry-After header and that interval respected rather than overriding it with your own backoff schedule.

Set a hard cap on retry count — three to five attempts is typical for synchronous request paths. Beyond that, open a circuit breaker. A retry budget approach limits the total fraction of requests that are retries (e.g., no more than 10% of all outgoing calls may be retries), preventing runaway retry amplification under sustained load. Libraries like Resilience4j (Java), Polly (.NET), and tenacity (Python) provide built-in retry policies with configurable predicates, backoff strategies, and attempt limits. AWS SDK clients implement adaptive retries by default, which adjust retry rate based on observed error rates.

For messaging systems, retries are typically handled at the broker level via re-delivery policies; do not add application-layer retries on top without understanding the combined effect on delivery count and DLQ routing.

Common failure modes

  • Retrying non-idempotent operations: duplicate payments, duplicate database inserts, duplicate emails sent to customers — the most damaging retry bug in production.
  • No error classification: retrying on 400, 401, or 404 which will never succeed, wasting latency budget.
  • Retry storms: hundreds of clients simultaneously failing and retrying at the same interval, causing the target to remain overloaded indefinitely.
  • Missing circuit breaker: retries continue indefinitely during a prolonged outage, saturating thread pools and cascading to the caller's callers.
  • Ignoring Retry-After header: hammering a rate-limited API every 100 ms instead of respecting the backoff hint it provides.

Decision checklist

  • Is every operation protected by this retry policy idempotent? Have you verified no duplicate side effects can occur?
  • Does your error classifier distinguish transient (retryable) from permanent (non-retryable) errors by HTTP status code and exception type?
  • Is the maximum retry count bounded (e.g., 3–5 attempts for synchronous paths)?
  • Is exponential backoff with jitter applied to avoid synchronized retry waves?
  • Is a circuit breaker wired after the retry policy to halt retries during sustained outages?
  • Are retry metrics (retry rate, retry success rate) exposed and alerted on to surface hidden problems?
  • Does your retry policy respect Retry-After headers from rate-limited upstream APIs?

Example use cases

  • An e-commerce checkout service retrying a payment gateway call that returned 503 during a brief gateway restart — the charge is idempotent because an idempotency key is sent with every request.
  • A microservice retrying a database read that timed out during a brief primary failover in an RDS Multi-AZ configuration.
  • An AWS Lambda function retrying an S3 PutObject call that failed with a 500 error from the S3 regional endpoint under burst load.
  • A gRPC client retrying a UNAVAILABLE status code from a service mesh sidecar when the upstream pod was momentarily unready during a rolling deploy.
  • Exponential Backoff with Jitter — the delay strategy that should accompany every retry policy.
  • Circuit Breaker — halts retries once failure rates exceed a threshold, preventing retry storms.
  • Bulkhead Pattern — isolates retry thread pools so a retry storm against one dependency cannot exhaust shared resources.
  • Idempotency — the safety property that makes retries safe to apply.

Further reading