Request Throttling

What it is

Request throttling is the server-side practice of controlling the rate at which incoming requests are admitted for processing, typically by queuing excess requests or signaling backpressure to clients rather than immediately rejecting them. While rate limiting counts requests over a window and rejects excess ones, throttling focuses on the processing rate — ensuring the server never accepts more work than it can handle at a given moment. Throttling is admission control at the ingress: it regulates the flow of work into a system to prevent overload.

Key mechanisms include: concurrency limiting (limiting the number of requests being processed simultaneously, not just the rate), queue-based throttling (excess requests wait in a queue bounded by a maximum depth), priority queuing (high-priority requests like health checks or premium users get served ahead of low-priority bulk operations), and adaptive throttling (automatically adjusting the admission rate based on server health metrics like CPU usage, queue depth, or downstream response times).

Why it exists

A server has a finite concurrency capacity — a certain number of threads, goroutines, or event loop slots. Accepting more concurrent requests than this capacity causes thread pool exhaustion, out-of-memory errors, or extreme latency increases as the OS scheduler thrashes between too many threads. Throttling prevents this by ensuring the concurrency stays within bounds even under overload. The key insight is that it is better to make some requests wait (finite queue) than to allow all requests to proceed slowly and fail due to resource exhaustion.

Queueing theory (Little's Law: L = λW) formalizes this: average concurrency L equals arrival rate λ multiplied by average processing time W. If λ exceeds your system's capacity, concurrency grows unbounded. Throttling caps λ at the ingress, keeping L within the system's safe operating zone. This is particularly important for protecting stateful dependencies like databases: a database running at 5ms query latency when handling 100 concurrent queries might run at 500ms under 1,000 concurrent queries as lock contention and I/O saturation compound. Throttling the application tier prevents cascading degradation in downstream tiers.

When to use

• Services with limited downstream capacity (databases, external APIs, GPU inference) that will degrade non-linearly under overload.
• CPU-bound or memory-bound workloads where accepting more concurrency than available cores leads to context-switch overhead and memory pressure.
• Batch processing APIs (file uploads, bulk imports) where a few large requests should not block all interactive user requests.
• Services that experience periodic traffic surges and prefer to delay requests briefly rather than fail them immediately.
• Microservices that must implement backpressure from overwhelmed downstream services.

When not to use

• Interactive user-facing endpoints where queuing adds latency that degrades user experience worse than an immediate 503. For interactive APIs, rejecting quickly with a clear retry signal is better than a silent 30-second queue wait.
• Stateless, fast API endpoints where the bottleneck is network throughput rather than server concurrency — queuing adds overhead without benefit.
• When clients cannot handle delayed responses (fixed HTTP timeouts shorter than maximum queue wait time).

Typical architecture

  Concurrency-Based Throttling (semaphore model):
  max_concurrency = 200
  active_requests = semaphore(200)

  on request_arrive:
    if active_requests.acquire(timeout=5s):
      process_request()
      active_requests.release()
    else:
      return HTTP 503 (queue full / timeout)

  Priority Queue Throttling:
  Incoming requests → classify by priority
    HIGH: health checks, premium users, reads  → queue_high
    NORMAL: standard API calls                 → queue_normal
    LOW: bulk exports, analytics, background   → queue_low

  Workers drain HIGH first, then NORMAL, then LOW
  → LOW-priority requests are shed first during overload

  Adaptive Throttling (Google SRE model):
  client_side throttle:
    requests_accepted   = count of accepted requests
    requests_total      = count of all requests
    K = 2.0  (multiplier, default 2)

    throttle_prob = max(0, (requests_total - K * requests_accepted)
                           / (requests_total + 1))

  → Clients self-throttle: they make fewer requests when the
    server is rejecting many of them, reducing server load.

  NGINX Request Throttling:
  limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;
  server {
    location /api/ {
      limit_req zone=api burst=200 nodelay;
      # burst=200 allows 200 queued requests
      # nodelay: burst requests processed immediately, not delayed
    }
  }

  gRPC Flow Control:
  Client sends WINDOW_UPDATE frames to control how much data
  server can send. On overload, server stops issuing WINDOW_UPDATEs
  → backpressure flows back to the calling client automatically.
          

Pros and cons

Implementation notes

Concurrency vs. rate limiting: These are complementary controls with different semantics. Rate limiting says "no more than N requests per second." Concurrency limiting says "no more than N requests active simultaneously." A server with 10 threads can handle 10 concurrent 1-second requests = 10 req/s OR 10 concurrent 10-millisecond requests = 1,000 req/s. Rate limiting alone doesn't protect against long-running requests that hold threads. Always implement both: rate limiting prevents per-client abuse; concurrency limiting prevents total server overload regardless of client behavior.

Queue depth and timeout: The queue depth determines how much latency smoothing you provide vs. how much memory overhead you accept. A queue depth of 2× your concurrency limit is a reasonable starting point. More importantly, set a queue timeout equal to your client's expected timeout minus your processing time. If clients time out after 5 seconds and processing takes 1 second, queue timeout should be 4 seconds. Requests that wait longer than that will fail on the client side anyway — there is no point completing them.

Common failure modes

• Queue buildup under sustained overload: Traffic exceeds capacity for minutes; the queue fills, requests stack up with increasing wait times. Without a queue cap, memory is exhausted. Always cap queue depth and fail fast when the queue is full.
• Head-of-line blocking: A few slow requests at the front of the queue block all subsequent requests regardless of their own processing time. Use parallel processing (thread pools, goroutines) so slow requests don't block fast ones. FIFO queues are particularly vulnerable; consider concurrent processing with a semaphore instead.
• Thundering herd on queue drain: After a bottleneck clears, all queued requests are released simultaneously. Use a leaky bucket or token bucket on the output side of the queue to smooth the drain rate.
• Downstream overload via queue release: Throttling the API tier without throttling downstream calls means a queue release floods the database. Throttling must be applied at each tier independently.

Decision checklist

• Maximum safe concurrency for the service has been established via load testing, not guessing.
• Queue depth and timeout are defined based on client timeout expectations and memory budget.
• Priority tiers are defined for different request classes (health checks, user-facing, bulk/background).
• 503/429 responses on queue full include Retry-After to guide client retry timing.
• Throttling is applied independently at each tier (API, application, database) — not just at the edge.
• Backpressure signals are propagated to upstream callers rather than silently queuing indefinitely.

Example use cases

• Batch file processing API: An API that converts uploaded documents throttles concurrent processing to 50 files, regardless of how many are submitted. A submission backlog of up to 200 files is queued; beyond that, new uploads receive 503 with Retry-After: 60. Interactive document preview requests bypass the batch queue via priority queuing.
• Database-backed microservice: A product search service has a database connection pool of 100 connections. The service uses a semaphore of 100 to limit concurrency, ensuring it never sends more than 100 concurrent queries to the database. Under heavy load, the semaphore queue holds up to 50 additional requests briefly while concurrent queries complete.
• gRPC streaming service: A data export service uses gRPC server-side streaming with flow control. When the client's receive buffer is full (slow client), the server pauses sending automatically via HTTP/2 flow control windows, preventing unbounded memory accumulation in the server's send buffer.

Related patterns

• Rate Limiting — rate limiting rejects excess requests; throttling queues them. Both protect server resources from overload.
• Connection Pooling — the database connection pool is itself a concurrency-limiting throttle; its size directly determines safe concurrency.
• Bulkhead Pattern — bulkheads isolate concurrency pools per tenant or feature; throttling limits total concurrency across all pools.

Further reading

• Google SRE Book — Handling Overload (Chapter 21)
• NGINX — ngx_http_limit_req_module Documentation
• gRPC — Flow Control