Observability & Operations

SLOs and Error Budgets

SLI/SLO/SLA hierarchy, error budget calculation, multi-window burn rate alerts for fast and slow budget consumption, and the Google SRE approach to balancing reliability with feature velocity.

⏱ 11 min read

What it is

SLOs (Service Level Objectives) are quantitative reliability targets that define the expected level of service for users. The SLI/SLO/SLA hierarchy consists of three distinct concepts that are frequently conflated: A Service Level Indicator (SLI) is a carefully defined quantitative measure of some aspect of service quality — it is the thing being measured. Examples: ratio of successful requests to total requests; proportion of requests served in under 200ms; successful storage writes. An SLO is a target value or range for an SLI: "99.9% of requests in the past 30 days should return 2xx within 300ms." An SLO is an internal target agreed between engineering and product. A Service Level Agreement (SLA) is a commercial contract that includes financial penalties for SLO violations — SLAs are typically a subset of SLOs, set lower than internal targets to create a buffer. Breaking an SLO is an engineering signal. Breaking an SLA has business consequences.

The error budget is the complement of the SLO: if a service has a 99.9% availability SLO, the error budget is 0.1% of requests over 30 days, equivalent to 43.2 minutes of total downtime or 432 failed requests per 100,000. The error budget is not a penalty — it is a budget that can be intentionally spent. Spending error budget on risky deployments is a deliberate engineering decision; spending it unintentionally on incidents is a signal that reliability investment is needed. When error budget is healthy (remaining), teams can invest in feature development. When error budget is nearly exhausted, teams should halt risky changes and invest in reliability improvements. This creates a data-driven mechanism for negotiating between development velocity and reliability.

Multi-window burn rate alerting alerts not on raw error rate but on the rate at which the error budget is being consumed. A burn rate of 1 means budget is being consumed exactly as expected. A burn rate of 14 means budget is being consumed 14× faster than normal — at this rate, the entire 30-day budget would be exhausted in approximately 2 days. The multi-window approach (alert only when both a short window and a long window exceed the threshold) reduces false positives from transient spikes.

Why it exists

Traditional approaches to reliability ("five nines") create perverse incentives: engineers pursue unmeasurable perfection, product managers have no data for release decisions, and incidents are measured inconsistently. SLOs provide a common language between engineering, product, and business: "we have 23 minutes of error budget remaining this month; this risky deployment should wait until next month" is a conversation grounded in data. The Google SRE book introduced error budgets as a tool for managing the tension between "developers want to deploy features" and "SREs want to maintain stability" — the budget is objective, owned by both, and creates aligned incentives.

When to use

  • Services with user-facing reliability expectations — SLOs make those expectations explicit and measurable.
  • Teams wanting data-driven decisions about release velocity vs reliability investment.
  • Organisations scaling SRE practices — SLOs are the foundation of the SRE model.

When not to use

  • Internal tooling or batch jobs where user impact is indirect — SLOs are most valuable for services where reliability directly affects user experience; for internal tools, simpler availability monitoring may suffice.

Typical architecture

SLO DEFINITION:
  Service: Checkout API
  SLI: ratio of requests returning 2xx within 500ms
  SLO: 99.9% of requests over rolling 30 days
  Error budget: 0.1% = 43.2 min/month = 0.1% of requests

SLI MEASUREMENT (Prometheus recording rules):
  # Good requests
  - record: job:checkout_good_requests:rate5m
    expr: sum(rate(http_requests_total{
              job="checkout", status=~"2..", le="0.5"}[5m]))

  # Total requests
  - record: job:checkout_total_requests:rate5m
    expr: sum(rate(http_requests_total{job="checkout"}[5m]))

  # Error ratio
  - record: job:checkout_error_ratio:rate5m
    expr: 1 - (job:checkout_good_requests:rate5m
                / job:checkout_total_requests:rate5m)

BURN RATE CALCULATION:
  SLO = 0.999
  Error rate threshold = 1 - 0.999 = 0.001

  Burn rate = current_error_rate / error_rate_threshold

  burn_rate:checkout:1h = avg_over_time(
    job:checkout_error_ratio:rate5m[1h]) / 0.001

MULTI-WINDOW BURN RATE ALERTS:
  # Fast burn: exhausts 30d budget in ~2d (14× rate)
  # Alert: both 1h AND 5m windows elevated
  P1: burn_rate:1h > 14 AND burn_rate:5m > 14

  # Moderate burn: exhausts 30d budget in ~5d (6× rate)
  # Alert: both 6h AND 30m windows elevated
  P2: burn_rate:6h > 6 AND burn_rate:30m > 6

  # Slow burn: exhausts 30d budget in ~15d (3× rate)
  # Ticket: burn_rate:3d > 3 sustained for 1h

ERROR BUDGET TRACKING DASHBOARD:
  - Current month error budget remaining (%)
  - Budget burn rate (current vs target)
  - Projected budget exhaustion date at current rate
  - Historical incidents contributing to budget spend
  - Budget status: HEALTHY / CAUTION / CRITICAL / FROZEN

Pros and cons

Pros

  • Error budgets create objective, data-driven conversations about deployment risk — "we have 8% budget remaining" is a harder argument to dismiss than "I feel this is too risky."
  • SLOs align engineering incentives toward user-visible reliability rather than internal metrics that may not correlate with user experience.
  • When error budget is healthy, teams have explicit permission to deploy risky changes and invest in features — this prevents the "always cautious" behaviour that slows development velocity unnecessarily.

Cons

  • SLOs require careful SLI definition — measuring the wrong thing (availability of health endpoint vs availability of business transactions) creates false confidence; SLI definition requires understanding of what users actually care about.
  • Setting initial SLO targets is difficult — too tight and the budget is constantly exhausted; too loose and it provides no signal. Teams should start with data-derived targets (current performance - buffer) and tighten over time.
  • Organisational adoption is challenging; "the budget says we should freeze deployments" requires management buy-in to have teeth rather than becoming a suggestion teams route around under pressure.

Implementation notes

Start with one or two critical user journeys (checkout, login, core API) rather than defining SLOs for every service simultaneously. For each journey, define the SLI as a ratio of "good events" to "total events" where "good" is defined from the user's perspective (correct response, within latency target). Derive the initial SLO target from historical performance: measure the actual error ratio over the last 90 days, then set the SLO slightly below the observed worst month (e.g., if 30-day error ratio has never exceeded 0.08%, set SLO at 99.9% to start). This prevents a situation where the SLO is breached from day one.

Integrate the error budget into the deployment process: add an error budget status check to the CI/CD pipeline or release process. When budget is critical (<10% remaining), automated checks should require explicit override with written justification. This enforcement mechanism is what gives SLOs their value — without it, the SLO is advisory only. Track error budget spend by cause (incidents, planned maintenance, bad deployments) to identify where to invest reliability effort.

Common failure modes

  • SLO too tight: Setting a 99.99% SLO for a service that historically achieves 99.5% means the budget is always exhausted, the signal is meaningless, and teams are always "in crisis" — desensitising them to budget state.
  • Wrong SLI measurement: Measuring availability of the health check endpoint (which is always up) rather than actual user transaction success rates provides false assurance of reliability.
  • No enforcement: Error budget exhausted but deployments continue anyway with no consequence — the budget becomes advisory and loses its value as a decision-making tool.
  • 100% SLO target: 100% availability is mathematically impossible; setting a 100% SLO means any incident at all exhausts the budget, creating an incentive to hide or minimise incidents rather than respond to them honestly.

Decision checklist

  • Are SLIs defined from the user's perspective (correct, timely responses) not internal metrics?
  • Are initial SLO targets derived from historical performance data?
  • Is the error budget calculated and tracked on a visible dashboard?
  • Are multi-window burn rate alerts configured for P1/P2 thresholds?
  • Is there an enforcement mechanism linking budget state to deployment decisions?
  • Are error budget spend incidents tracked by cause to identify reliability investment priorities?

Example use cases

  • SLO-driven release freeze: Payment service has 99.95% SLO; a production incident consumes 70% of monthly budget; engineering freezes risky deployments for remainder of month; product team accepts the trade-off given objective data rather than ongoing negotiation.
  • Multi-service SLO hierarchy: Mobile app defines a product-level SLO (user checkout success rate) composed of SLOs for cart, payment, and fulfillment services; each service team owns its contribution to the product SLO; shared visibility creates cross-team alignment on reliability priorities.
  • Healthy budget enables innovation: Authentication service has 2× the expected error budget remaining through the quarter; team explicitly uses this to justify a risky database migration; migration causes a 30-minute degradation consuming 40% of remaining budget, which was planned and accepted based on the budget state.

Further reading