Alerting Strategies

What it is

Alerting is the mechanism that notifies engineers when systems require human attention. Effective alerting is not about detecting every anomaly — it is about detecting every situation that requires a human to act, and nothing else. Alert fatigue occurs when on-call engineers receive too many alerts, most of which are false positives, low-urgency, or self-resolving. The result is desensitisation — engineers learn to dismiss alerts, and real incidents are missed or responded to slowly. Google's SRE philosophy defines a good alert as one that is urgent, actionable, and novel: urgent (requires response now), actionable (the responder can do something meaningful), and novel (not already handled by automation).

Symptom-based alerting alerts on user-visible impact — high error rate, high latency, service unavailability. This is preferred because it aligns with user experience: if users are not impacted, an alert should not wake up an engineer. Cause-based alerting alerts on potential causes before they cause user impact — disk 80% full, CPU at 90%, memory pressure. Cause-based alerts are useful for predictive capacity management but should not be paging alerts; they should be tickets or low-urgency notifications. The majority of paging alerts should be symptom-based.

SLO-based alerting with error budget burn rate is the most sophisticated and effective approach. Rather than alerting on a fixed threshold (error rate > 1%), burn rate alerts ask: "at the current error rate, how quickly are we consuming our 30-day error budget?" A fast burn rate (consuming the budget 14× faster than normal) warrants an immediate page. A slow but sustained elevated burn rate (3× normal over 6 hours) warrants a ticket or lower-urgency notification. This approach naturally prioritises alerts by user impact — a brief 5% error rate that lasts 10 minutes consumes little error budget; a sustained 0.5% error rate over 48 hours consumes far more.

Why it exists

Naive alerting (alert on any anomaly, page on every threshold breach) is common and dysfunctional. Teams with high alert volumes train themselves to ignore alerts — this is documented as "alarm fatigue" in both IT operations and medical literature. The consequences are missed incidents, engineer burnout, poor on-call quality of life, and ultimately turnover. Principled alerting design, grounded in SLO-based burn rate analysis, produces alerting systems that page only when user impact is occurring or imminent, dramatically reducing alert volume while improving mean time to detection (MTTD) for real incidents.

When to use

• All production systems with on-call engineers — effective alerting design is a reliability and quality-of-life requirement.
• Systems with defined SLOs — SLO-based burn rate alerting is the recommended approach for services with explicit reliability commitments.
• Teams experiencing alert fatigue — a structured alert audit and redesign is a high-value reliability investment.

When not to use

• Not applicable — all teams with production systems need some alerting strategy. The question is only which approach.

Typical architecture

ALERTING ARCHITECTURE:

Prometheus/Grafana → AlertManager → PagerDuty/Opsgenie

ALERT SEVERITY LEVELS:
  P1 (CRITICAL): User-facing impact now → Page immediately
    Response: <15 minutes, 24x7
    Example: Checkout error rate > 5%

  P2 (HIGH): Degraded experience / fast burn → Page
    Response: <1 hour, business hours
    Example: p99 latency > 2× SLO

  P3 (MEDIUM): Potential future impact → Ticket/Slack
    Response: next business day
    Example: Disk > 80%, slow burn rate

  P4 (LOW): Informational → Dashboard/report only
    Example: Non-critical batch job delayed

SLO-BASED BURN RATE ALERTS (Prometheus):
  # SLO: 99.9% availability (0.1% error budget over 30d)
  # Budget: 43.2 min/month = 0.1% × 30d × 24h × 60m

  # Fast burn (P1): consuming budget 14× faster
  # → Would exhaust 30d budget in ~2d
  # Fires if 1h error rate AND 5m error rate both elevated
  - alert: HighFastBurnRate
    expr: |
      (
        job:http_errors:rate1h{job="checkout"}
        / job:http_requests:rate1h{job="checkout"}
      ) > (14 * 0.001)
      and
      (
        job:http_errors:rate5m{job="checkout"}
        / job:http_requests:rate5m{job="checkout"}
      ) > (14 * 0.001)
    labels:
      severity: critical
    annotations:
      summary: "Checkout error budget burning fast"

  # Slow burn (P2): consuming budget 3× faster over 6h
  - alert: SlowBurnRate
    expr: |
      (
        job:http_errors:rate6h / job:http_requests:rate6h
      ) > (3 * 0.001)
    for: 1h
    labels:
      severity: warning

PAGERDUTY ROUTING:
  High severity → PagerDuty → on-call engineer phone
  Medium severity → PagerDuty → Slack #incidents channel
  Low severity → Slack #alerts channel (no page)

ALERT RUNBOOK:
  Each alert links to a runbook with:
  1. What this alert means
  2. Impact on users
  3. Diagnostic steps (what to check first)
  4. Mitigation steps (how to fix common causes)
  5. Escalation path

Pros and cons

Implementation notes

Start with an alert audit: list every alert, its last 30 days of firing history, and whether each firing required human action. Alerts that fired without requiring action are candidates for deletion or demotion to lower severity. Google recommends that 100% of pages should require human action — any alert that regularly fires without action should be eliminated. This audit alone typically reduces alert volume by 50% in teams that have accumulated alerts over years.

Every alert must have a runbook linked in the annotation. On-call engineers paged at 03:00 should be able to follow a runbook to diagnose and mitigate the most common causes within minutes, even without full context. Runbooks should be kept current — a runbook that describes a system that no longer exists is worse than no runbook, because it wastes investigation time. Treat runbooks as code: maintain them in version control, review changes with the same rigour as code changes, and run "fire drills" (test the runbook with a simulated incident) to validate they are accurate.

Common failure modes

• Alerting on causes rather than symptoms: Paging on CPU > 80% or memory > 85% when there is no user impact trains engineers to ignore alerts. Page on symptoms (latency, errors), file tickets on causes (resource pressure).
• No runbooks: Engineers receive an alert at 03:00 with no context on what the alert means or how to respond — investigation time is wasted on diagnosis that a runbook could eliminate.
• Alert storms during incidents: A single root cause triggers 50 dependent alerts; alert grouping and inhibition rules in AlertManager must suppress downstream alerts when the root cause alert is firing.
• Flapping alerts with no hysteresis: Alerts that fire and resolve every few minutes because the metric oscillates around the threshold; add a for duration (e.g., for: 5m) to require sustained breach before alerting.

Decision checklist

• Are all paging alerts symptom-based (user-facing impact)?
• Is every alert linked to a current runbook with diagnostic and mitigation steps?
• Are SLO-based burn rate alerts implemented for critical user journeys?
• Are alert severity levels defined with explicit response time SLAs?
• Is alert grouping/inhibition configured to suppress cascading alerts during incidents?
• Has an alert audit been performed in the last 6 months to eliminate noisy/unnecessary alerts?

Example use cases

• Alert fatigue remediation: Team receiving 200+ alerts/week performs audit; 60% are cause-based alerts that never required action (CPU, disk, queue depth within normal range); demoted to tickets or dashboards; paging alerts drop to 15/week, all actionable, on-call quality of life significantly improved.
• SLO burn rate alert catches slow degradation: A new dependency (third-party API) starts returning 5xx on 0.3% of requests — below any fixed threshold alert but consuming error budget at 3× the expected rate; slow burn alert fires after 6 hours; team investigates and finds the upstream degradation before it escalates.
• On-call routing by severity: P1 alert (checkout down) pages on-call mobile at any hour; P2 alert (elevated latency) sends Slack message to #incidents channel during business hours, pages after hours; P3 (disk at 80%) creates Jira ticket for next sprint.

Related patterns

• SLOs and Error Budgets — The foundation for burn rate alerting.
• Metrics and Monitoring — The data source for alert evaluation.
• Incident Response — What happens after an alert fires.

Further reading

• Google SRE Workbook: Alerting on SLOs
• Effective Monitoring and Alerting (O'Reilly)
• Prometheus AlertManager Docs