Reliability & Resilience Scalability

Load Shedding

Intentionally dropping low-priority requests under overload to preserve capacity for critical workloads.

⏱ 10 min read

What it is

Load shedding is the deliberate rejection of incoming requests when a system detects it is operating beyond its safe capacity. Rather than accepting all requests and degrading uniformly — increasing latency, exhausting resources, and eventually failing entirely — a system under overload identifies which requests are least critical and rejects them immediately with an HTTP 503 Service Unavailable (with a Retry-After header), preserving capacity for higher-priority workloads that must continue to succeed.

Load shedding is distinct from rate limiting: rate limiting enforces per-client quotas to prevent any single client from consuming disproportionate resources; load shedding is a global, server-side admission control mechanism that activates when overall system capacity is exceeded. The two are complementary — rate limiting prevents individual client abuse; load shedding protects the system during aggregate overload regardless of source.

Why it exists

A system that accepts all requests under overload will queue them, consuming memory, threads, and CPU for work that will likely time out before completion. As the queue grows, latency for all requests increases, including high-priority ones. Eventually, the system enters a state of collapse where it is fully occupied processing (or timing out) low-value work while high-value requests cannot get service at all. Load shedding prevents this collapse by deliberately keeping the system below its safe operating limit — accepting fewer requests but processing each one correctly within SLA rather than accepting all requests and processing none of them correctly.

When to use

Services that experience unpredictable burst traffic (viral events, flash sales, DDoS) where autoscaling cannot respond fast enough.
Systems with a meaningful priority split between workloads (user-facing vs. background, payment vs. analytics).
Any service that cannot horizontally scale fast enough to absorb sudden traffic spikes.
Services where the cost of total collapse under overload is higher than the cost of rejecting a fraction of requests.
Batch processing pipelines that mix real-time urgent tasks with best-effort background work on the same infrastructure.

When not to use

When all requests are equally critical and no priority ordering can be defined.
When the system can reliably autoscale fast enough to absorb bursts before overload occurs — prefer autoscaling to shedding.
Synchronous user-facing flows where a 503 response causes an unacceptable user experience — graceful degradation or queueing may be preferable.
When clients are not designed to handle 503 + retry-after responses gracefully — the behaviour of naive clients must be understood first.

Typical architecture

Load Shedding with Priority Admission Control
════════════════════════════════════════════════

Incoming Requests → Load Measurement → Admission Decision
                         │
              ┌──────────┴──────────┐
              │  Load Signals:       │
              │  • CPU utilisation   │
              │  • Queue depth       │
              │  • p99 latency       │
              │  • Active threads    │
              └──────────┬──────────┘
                         │ overload detected
                         ▼
         Priority Classification
         ┌─────────────────────────────┐
         │ P0: Health checks / payments│ ← always admit
         │ P1: User reads (browse)     │ ← shed if CPU > 90%
         │ P2: User writes (orders)    │ ← shed if CPU > 80%
         │ P3: Background sync / audit │ ← shed if CPU > 70%
         └─────────────────────────────┘
                         │
         ┌───────────────┴──────────────┐
         │ Admitted                      │ Rejected
         ▼                              ▼
   Process normally              HTTP 503
                                 Retry-After: 5
                                 (client backs off)

Pros and cons

Pros

Protects the system from total collapse under extreme load, preserving correct behaviour for admitted requests.
High-priority workloads (payments, health checks) continue to succeed even when overall request volume exceeds capacity.
A 503 with Retry-After is a well-understood HTTP semantic that enables client cooperation — well-behaved clients back off.
Provides a controllable, measurable safety valve for handling unexpected traffic bursts.
Adaptive load shedding can adjust thresholds dynamically as system capacity changes (e.g., during rolling restarts).

Cons

Rejecting requests is never ideal — affected users see errors, which damages experience and trust.
Priority classification is complex and must be maintained as the system evolves — new request types need classification.
Misconfigured thresholds can trigger shedding too aggressively (over-shedding) or not aggressively enough (under-shedding).
Clients that do not handle 503 gracefully may retry aggressively, worsening load rather than helping.
Does not address the root cause — if the service is regularly shedding load, it needs more capacity or efficiency improvement.

Implementation notes

Measure load using multiple signals simultaneously, not just CPU: CPU is a lagging indicator during I/O-bound workloads. Queue depth (length of the inbound request queue), active goroutine/thread count, and p99 response latency are often more accurate early warning signals. Combine signals with a weighted formula or use a hysteresis threshold (activate shedding at 80% load; deactivate only when load drops below 60%) to prevent rapid oscillation.

Priority assignment should be done at the entry point — API gateway or service mesh — where request metadata (HTTP method, path, tenant tier, user type) is most easily accessible. Use token bucket algorithms for priority-weighted admission: P0 traffic always gets tokens; P1 traffic gets tokens when the P0 bucket is not exhausted; P2 and P3 traffic get tokens only when surplus capacity remains. Always return HTTP 503 with a Retry-After header specifying the recommended wait in seconds, and include a X-Reason: load-shedding header for client observability.

Common failure modes

Shedding critical traffic: misclassifying a critical endpoint as low-priority and shedding it during the overload event that most needs it to succeed.
No client cooperation: clients ignore 503 responses and retry immediately, creating a retry storm that keeps the server permanently overloaded.
Threshold too conservative: shedding begins at 50% CPU, unnecessarily rejecting traffic during normal load spikes that autoscaling would handle.
Single signal threshold: using only CPU means I/O-bound overload (e.g., all threads blocked on slow DB) is not detected until CPU has actually spiked.
No Retry-After header: clients cannot determine when to retry, either retrying too soon or giving up too early.

Decision checklist

Have you defined priority tiers for all request types with explicit P0/P1/P2/P3 classifications?
Are load signals multi-dimensional (CPU + queue depth + latency) rather than single-metric?
Do all 503 responses include a Retry-After header with a meaningful backoff suggestion?
Are shedding events logged and exposed as metrics (shed_requests_total per priority tier)?
Have you tested client behaviour under 503 responses to confirm clients back off rather than storm?
Is the shedding threshold high enough to avoid false triggers during normal load spikes?
Is there a hysteresis gap between the activate and deactivate thresholds to prevent oscillation?

Example use cases

Google's distributed systems shed background replication and analytics requests when serving the critical user-facing search path under traffic spikes.
An API gateway that sheds webhook delivery calls (P3) and batch export requests (P2) when CPU exceeds 85%, while continuing to serve synchronous user API requests (P1) and payment processing (P0).
A Kafka consumer group with a priority queue that processes payment events before analytics events, dropping analytics messages during peak load to keep payment processing within SLA.

Graceful Degradation — disables features; load shedding rejects requests entirely.
Bulkhead Pattern — pre-allocates resource pools by workload type, a complementary isolation mechanism.
Circuit Breaker — shed at the client side; load shedding is the server-side equivalent.