Reliability & Resilience Essential

Circuit Breaker

Stopping calls to a failing dependency after a threshold to prevent cascading failures and allow recovery.

⏱ 11 min read

What it is

The Circuit Breaker pattern wraps calls to an external dependency and tracks the failure rate of those calls over a sliding window. When the failure rate exceeds a configured threshold, the circuit "trips" into the Open state and immediately returns an error (or invokes a fallback) for all subsequent calls without even attempting to reach the dependency. After a configured wait period, the circuit transitions to Half-Open, allowing a limited probe request through to test whether the dependency has recovered. If the probe succeeds, the circuit closes and normal traffic resumes; if it fails, the circuit opens again for another wait period.

The pattern was popularised by Michael Nygard in Release It! (2007) and is now a standard component in most resilience libraries. It differs fundamentally from the Retry pattern: retries assume the failure is transient and immediately try again, while circuit breakers assume the system is in sustained trouble and need to stop calling it entirely to allow recovery. The two patterns are complementary — retries handle brief transient blips; circuit breakers handle prolonged outages.

Why it exists

Without a circuit breaker, every call to a failing dependency blocks until timeout, consuming a thread or connection for the full timeout duration. Under load, all threads fill up waiting for a non-responsive service, causing the caller itself to become unresponsive — even though the caller's own code is not broken. This is cascading failure: one degraded downstream service takes down an entire call chain of otherwise healthy services.

Circuit breakers prevent this by failing fast: an open circuit returns immediately with an error, freeing the caller's thread in microseconds rather than seconds. This preserves the caller's thread pool and connection pool capacity, protecting it from being overwhelmed by a downstream failure it cannot control. The pattern also gives the failing dependency breathing room — a dependency under extreme load that is receiving zero traffic has the best chance of recovering and stabilising.

When to use

Calls to any external or cross-service dependency where the caller cannot afford to have its own threads consumed by timeouts.
Services with response time SLOs — an open circuit triggers an immediate fallback, preserving the caller's p99 latency even when the dependency is degraded.
High-traffic services where even 1% of requests timing out would saturate the thread pool within seconds.
Any service with a clearly defined fallback (cached data, default response, degraded mode) that can activate when the circuit opens.
Wrapping calls to third-party APIs, databases, or caches that are outside the team's direct control.

When not to use

Calls with very low traffic volume where a failure-rate threshold would need to be met by just one or two requests, causing false trips.
Operations where any error should immediately propagate to the user without retry or fallback (e.g., explicit user-initiated deletes).
When the failure mode is client-side (bad inputs, auth errors) rather than server-side — circuit breakers are for infrastructure failures, not input validation failures.
In-process function calls — the overhead is unjustified when there is no network or resource contention involved.

Typical architecture

Circuit Breaker State Machine
══════════════════════════════════════════════════════

              failure rate > threshold
  ┌──────┐   OR slow call rate > threshold   ┌──────┐
  │      │ ──────────────────────────────────►      │
  │CLOSED│                                   │ OPEN │
  │      │◄──────────────────────────────────       │
  └──────┘   success count > threshold       └──┬───┘
     ▲        (from Half-Open)                  │
     │                                          │ wait_duration_in_open_state
     │                                          ▼ (e.g. 60 seconds)
     │                                      ┌───────────┐
     │   failure in probe                   │           │
     └──────────────────────────────────────│ HALF-OPEN │
         success in probe ──────────────────│           │
                              (closes)      └───────────┘

Resilience4j Configuration Example:
────────────────────────────────────
  slidingWindowType: COUNT_BASED
  slidingWindowSize: 10
  failureRateThreshold: 50        # 50% failure rate trips breaker
  slowCallRateThreshold: 100      # 100% of calls taking > slowCallDurationThreshold
  slowCallDurationThreshold: 2s
  waitDurationInOpenState: 60s
  permittedNumberOfCallsInHalfOpenState: 3
  automaticTransitionFromOpenToHalfOpenEnabled: true

Pros and cons

Pros

Prevents cascading failures — an open circuit fails fast without consuming caller threads or connections.
Gives failing dependencies recovery time by removing all incoming load during the open period.
Enables fast fallback activation — the caller knows immediately that the dependency is unavailable and can serve cached or default data.
Provides real-time visibility into dependency health — an open circuit is an explicit alert that a dependency is in trouble.
Composable with retries, bulkheads, and fallbacks to form a comprehensive resilience stack.

Cons

Adds operational complexity — teams must monitor circuit state and tune thresholds per dependency.
False positives: a brief error spike can trip the circuit, unnecessarily blocking traffic even after the dependency has recovered.
Half-Open state is a single point of re-evaluation — a single slow probe response can keep the circuit open longer than necessary.
Requires a meaningful fallback strategy; without one, an open circuit just converts slow failures into fast failures, which may still break the user experience.
Sliding window configuration is non-trivial for low-traffic services where sample sizes are small.

Implementation notes

Resilience4j (Java) is the modern successor to Hystrix and supports both count-based and time-based sliding windows, slow-call detection, and automatic Half-Open transitions. Configure a separate circuit breaker instance per downstream dependency rather than sharing one, so that a failure in the payment service does not affect the product catalogue circuit breaker. The slowCallDurationThreshold is as important as the error rate threshold: a dependency that responds in 30 seconds is effectively unavailable even if it technically returns 200 OK.

Always pair the circuit breaker with a fallback. When the circuit is open, invoke the fallback synchronously — return cached data, a default empty list, a stub response, or an appropriate error that the UI can handle gracefully. The fallback should itself be simple and fast; do not call another network dependency from within the fallback. Export circuit state as a metric (resilience4j.circuitbreaker.state in Prometheus) and alert on transitions to Open state — each Open event is a signal that a dependency is unhealthy and may need operator attention.

Common failure modes

Sharing one circuit breaker across all dependencies: a failure in Service A trips the breaker and blocks calls to the healthy Service B and Service C.
Threshold too low: a brief traffic spike with a few timeouts trips the circuit, causing unnecessary service degradation during otherwise healthy operation.
No fallback defined: an open circuit returns a raw exception to the user instead of graceful degraded content.
Not monitoring circuit state: circuits trip and remain open undetected, silently degrading functionality for extended periods.
Half-Open probe not representative: the probe request type does not reflect the actual failing request type, leading to false recovery.

Decision checklist

Is a separate circuit breaker instance configured for each distinct downstream dependency?
Are both failure rate and slow-call rate thresholds configured (not just one)?
Is a fallback defined that will activate when the circuit opens?
Is circuit state exported as a metric and alerting configured on Open transitions?
Is the sliding window large enough to represent meaningful statistical signal for your traffic volume?
Is the wait duration in Open state long enough to give the downstream service time to recover?
Are timeout values set on the underlying HTTP/gRPC client to prevent indefinite blocking even when the circuit is Closed?

Example use cases

An order service wrapping its call to a payment gateway — if the gateway degrades, the circuit opens, calls return a "payment temporarily unavailable" fallback, and the thread pool is protected from timeout saturation.
A recommendation service wrapped by a circuit breaker in the product page API — when the recommendation ML service is slow, the circuit opens and the page loads with "no recommendations available" instead of timing out.
A service mesh (Istio, Linkerd) implementing circuit breaking at the proxy layer — outlier detection in Envoy can eject unhealthy endpoints automatically without application code changes.

Retry Pattern — the first layer of resilience; retries should feed into a circuit breaker.
Bulkhead Pattern — isolates thread pools so a tripped circuit breaker doesn't exhaust shared threads.
Fallback Pattern — what to call when the circuit is open.
Exponential Backoff — the delay strategy applied to retries before the circuit breaker trips.