API Design Scalability

API Rate Limiting

Protecting APIs from overload and abuse — rate limit algorithms (fixed window, sliding window, token bucket, leaky bucket), standard response headers, 429 handling with Retry-After, per-consumer and per-endpoint limits, burst allowance, and communicating limits in OpenAPI.

⏱ 12 min read

What it is

Rate limiting controls how many requests an API client can make within a defined time window, protecting the service from overload, abuse, and denial-of-service. Rate limiting is applied by counting requests attributed to an identity key (API key, user ID, IP address, or OAuth client) and enforcing a threshold.

Rate limiting operates at two conceptually distinct levels: throttling (slowing down a client that exceeds the rate) and hard rejection (returning 429 Too Many Requests). Most production APIs use hard rejection. Rate limiting is typically implemented at the API gateway or as a middleware layer, using a shared counter store (Redis) to coordinate across multiple instances.

Why it exists

Without rate limiting, a single misbehaving client (buggy retry loop, malicious crawler, DDoS) can consume all available server capacity and starve legitimate traffic. Rate limiting also enforces commercial tiering (a free plan gets 100 req/min; paid plans get 1,000 req/min) and prevents credential stuffing attacks by limiting auth endpoint call frequency.

Rate limiting algorithms

Fixed window counter

Window: 60s, Limit: 100 requests
Counter: { "user-42:2024-04-15T09:00:00": 47 }

At 09:00:59: request #48 → ALLOW (counter++)
At 09:01:00: new window starts, counter resets to 0
At 09:00:55: request #100 → ALLOW
At 09:00:56: request #101 → REJECT (429)
At 09:01:00: all 100 requests allowed again

Problem: A client can make 100 requests at 09:00:59 and another 100 at 09:01:01 — effectively 200 requests in 2 seconds around the window boundary. This "burst at boundary" flaw makes fixed window unsuitable for strict enforcement.

Sliding window log

Store a sorted set of request timestamps per client:
  ZADD user-42:timestamps {current_time} {request_id}
  ZREMRANGEBYSCORE user-42:timestamps 0 {current_time - 60}
  count = ZCARD user-42:timestamps
  if count >= 100: REJECT
  else: ALLOW

The sliding window log stores every request timestamp and counts only those within the last window period. Accurate but memory-intensive — storing timestamps for millions of clients is expensive.

Sliding window counter

rate = (prev_window_count * (1 - elapsed_fraction)) + current_window_count
if rate >= limit: REJECT

Example (60s window, limit 100):
  prev window: 80 requests
  current window started 30s ago (elapsed_fraction = 0.5)
  current window: 30 requests
  rate = (80 * 0.5) + 30 = 70 → ALLOW

The sliding window counter approximates the sliding log using two fixed-window counters. Low memory, accurate to within ~0.5% of the sliding log. This is the algorithm used by Cloudflare and most production implementations.

Token bucket

Bucket: capacity=100 tokens, refill_rate=10 tokens/second
State: { tokens: 45, last_refill: 09:05:00 }

At 09:05:03 (request arrives):
  elapsed = 3s → tokens_to_add = 3 * 10 = 30
  tokens = min(100, 45 + 30) = 75
  request costs 1 token → tokens = 74 → ALLOW

If tokens < 1: REJECT

Token bucket supports burst: a client that hasn't used the API for a while accumulates tokens (up to bucket capacity) and can burst above the average rate. This is the most client-friendly algorithm for APIs with variable workloads. AWS API Gateway and many CDNs use token bucket.

Leaky bucket

The leaky bucket enforces a strict output rate: requests enter a queue and are processed at a fixed rate (e.g., 10 req/s). Requests exceeding queue capacity are rejected. This smooths output to the backend at a constant rate regardless of input spikes. Used for backend protection (preventing thundering herds) rather than client-facing limits.

Response headers

Successful response (standard RateLimit headers, RFC draft):
HTTP/1.1 200 OK
RateLimit-Limit: 100
RateLimit-Remaining: 47
RateLimit-Reset: 1714000060    (Unix timestamp of window reset)
RateLimit-Policy: 100;w=60    (100 requests per 60-second window)

GitHub-style (widely adopted de facto):
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4987
X-RateLimit-Reset: 1714000060
X-RateLimit-Used: 13
X-RateLimit-Resource: core

Rate limit exceeded (429):
HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 30              (seconds until the limit resets)
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1714000060

{
  "type": "https://api.example.com/errors/rate-limited",
  "title": "Too Many Requests",
  "status": 429,
  "detail": "You have exceeded the rate limit of 100 requests per minute.",
  "retryAfter": 30
}

Include rate limit headers on every response, not just on 429. This allows clients to monitor their consumption and implement proactive throttling before hitting the limit. The IETF RateLimit headers (draft-ietf-httpapi-ratelimit-headers) are the standardising effort; use them alongside the widely deployed X-RateLimit-* variants for compatibility.

Limit dimensions

Per consumer — by API key, OAuth client ID, or authenticated user. Most common. Each consumer gets their own quota.
Per IP — for unauthenticated endpoints (auth, public). Caution: NATs and proxies may share a single IP among many legitimate users.
Per endpoint — expensive endpoints (search, export, PDF generation) have stricter limits than cheap list endpoints.
Per plan/tier — free, basic, professional, enterprise tiers with progressively higher limits. Gate by API key metadata.
Global — a hard ceiling across all consumers, protecting the service from aggregate overload.

Burst allowance

A burst allowance lets clients exceed their average rate for a short period. With a token bucket, bucket capacity defines the maximum burst. Example: 1,000 req/min rate with a 200-request burst capacity allows a client to fire 200 requests immediately after being idle, then refills at ~16 requests per second.

Document both the sustained rate and the burst limit. The Stripe API documents this as: "In live mode, you can make up to 100 reads and 100 writes per second in test mode; your account may have higher limits."

OpenAPI documentation

paths:
  /orders:
    get:
      x-rate-limit:
        limit: 1000
        window: 60
        unit: requests_per_minute
      responses:
        "200":
          headers:
            RateLimit-Limit:
              schema: { type: integer }
              description: Maximum requests allowed per window.
            RateLimit-Remaining:
              schema: { type: integer }
              description: Requests remaining in the current window.
            RateLimit-Reset:
              schema: { type: integer, format: unix-time }
              description: Unix timestamp when the window resets.
        "429":
          description: Rate limit exceeded
          headers:
            Retry-After:
              schema: { type: integer }
              description: Seconds to wait before retrying.
          content:
            application/problem+json:
              schema:
                $ref: "#/components/schemas/ProblemDetails"

Client implementation guidance

Well-behaved API clients should: (1) read RateLimit-Remaining on every response and slow down as it approaches zero; (2) on 429, read Retry-After and wait exactly that duration before retrying — do not implement fixed-interval retry; (3) use exponential backoff with jitter when Retry-After is not provided; (4) implement a local rate limiter that pre-throttles to stay within the limit rather than relying on 429 as the signal.

Pros and cons

Pros

Protects backends from overload and prevents a single consumer from starving others.
Enables fair resource allocation across consumers and pricing tiers.
Mitigates credential stuffing, brute force, and scraping attacks.
Token bucket burst support makes limits feel less restrictive for bursty legitimate workloads.

Cons

Distributed enforcement requires a shared counter store (Redis); adds latency and a single point of coordination.
Fixed window allows boundary bursts that can exceed intended limits by 2×.
Per-IP limits break behind NATs and shared egress proxies, penalising legitimate users.
Limits that are too tight cause legitimate users to build complex retry logic; communicate limits clearly.

Decision checklist

Are rate limit headers included in every response (not just on 429)?
Is Retry-After included on 429 responses?
Are per-endpoint limits defined for expensive operations (search, bulk, export)?
Is the rate limit algorithm appropriate — sliding window for accuracy, token bucket for burst support?
Is the counter store (Redis) highly available — does a Redis outage cause rate limiting to fail open or closed?
Are rate limits documented in the OpenAPI spec and developer portal?
Are auth endpoints and password-reset endpoints separately rate-limited to prevent brute force?

API Gateway — rate limiting is typically enforced at the gateway layer.
API Authentication — consumer identity is the dimension for rate limit attribution.
OpenAPI Specification — document rate limit headers and 429 responses.