Runbook Template

What it is

A runbook is an operational reference document that contains the step-by-step procedures an engineer needs to operate, troubleshoot, and remediate a specific service or system. It is the on-call engineer's first line of reference when an alert fires at 2am — a concrete, verified set of actions they can execute to restore service without requiring deep prior knowledge of the system's internals.

Runbooks range from simple single-procedure documents ("how to restart the worker pool") to comprehensive operational guides covering the full surface area of a complex service. The best runbooks are written by the engineers who built the system and continuously updated by the engineers who operate it, incorporating lessons from each incident and on-call rotation.

Modern runbooks increasingly include automation hooks — links to runnable scripts, Jupyter notebooks, or platform tooling that can execute remediation steps automatically. The document serves as both the human-readable guide and the index to automated tooling, ensuring that operators can either run steps manually or trigger automation as confidence grows.

Why it exists

Operational knowledge is perishable and unevenly distributed. Without runbooks, the ability to respond to incidents is gated on the availability of the engineer who last touched the affected system. This creates dangerous single points of failure in on-call rotations: when that engineer is unavailable, incident resolution times increase dramatically and junior engineers operate without guidance in high-pressure situations.

Runbooks transfer operational knowledge from individuals to the team. When an experienced engineer writes down the procedure for a complex remediation, that knowledge becomes accessible to every on-call engineer — including those who joined after the system was built. This reduces mean time to resolution (MTTR), decreases on-call stress, and makes it safer to rotate more engineers through on-call duty.

Runbooks also drive system improvement. The act of writing a runbook for a complex or multi-step procedure frequently reveals that the procedure could be simplified, automated, or eliminated through better system design. A runbook that requires fifteen manual steps to execute is a candidate for automation; a runbook that is invoked weekly suggests a systemic issue worth fixing.

When to use

• For every alert that pages on-call engineers — each alert should have a corresponding runbook entry that guides the responder through investigation and remediation.
• For complex operational procedures that require specific steps in a specific order, such as database failovers, key rotation, or traffic migration.
• For procedures that are infrequent enough that the engineer executing them may not remember the steps, such as quarterly certificate renewals or annual disaster recovery drills.
• When onboarding new engineers to on-call rotations — runbooks are a key component of on-call readiness training.
• For any procedure where doing steps in the wrong order or skipping a step can cause data loss, outage extension, or security exposure.

When NOT to use

• As a substitute for fixing the underlying cause of a recurring issue — a runbook for a problem that should be automated or eliminated is a known technical debt that should be scheduled for resolution.
• For procedures so trivial that any engineer could execute them without guidance (restarting a container, checking a log file) where documentation adds more overhead than value.
• As the sole repository of architectural knowledge — runbooks describe operational procedures, not system design. Use design docs and ADRs for architectural context.
• When the runbook would be more dangerous than helpful — if a procedure is complex enough that following it incorrectly could cause more harm, it should require human escalation rather than execution by an on-call responder.

Template

# Runbook: [Service Name]

**Service:** [Service name and brief description]
**Version:** [Runbook version — e.g., v1.3]
**Owner:** [Team name]
**On-call rotation:** [Link to PagerDuty / OpsGenie rotation]
**Last reviewed:** [YYYY-MM-DD]
**Next review due:** [YYYY-MM-DD]  

---

## Purpose

[2–3 sentences describing the service, its role in the system, and its dependencies.]

**SLA / SLO:** [e.g., 99.9% availability, p99 latency < 200ms]
**Business impact of outage:** [e.g., "Users cannot complete checkout; ~$5,000/minute revenue impact"]

---

## Prerequisites

### Access Requirements

- [ ] AWS Console access (role: `[role-name]`)
- [ ] Kubernetes cluster access: `kubectl config use-context [cluster-name]`
- [ ] Database read/write access (request via [access request tool])
- [ ] PagerDuty / incident channel: [#incident-channel-name]

### Key Dashboards
| Dashboard | URL | Purpose |
|-----------|-----|---------|
| Service overview | [URL] | Request rate, error rate, latency |
| Infrastructure | [URL] | CPU, memory, pod health |
| Database | [URL] | Query latency, connection pool |
| Business metrics | [URL] | Orders/minute, conversion rate |

### Key Repositories and Documentation
- Source code: [URL]
- Architecture overview: [Link to design doc or wiki]
- Deployment pipeline: [Link to CI/CD config]

---

## Architecture Overview

[Short description of the service's components and how they interact.
Example: "The Order Service consists of an HTTP API layer (3 pods),
a background worker (2 pods), a PostgreSQL primary, and a read replica.
It publishes to the `orders` Kafka topic consumed by Inventory and Notifications."]

**Critical path:** [e.g., Client → Load Balancer → API Pod → PostgreSQL Primary]

---

## Runbook Procedures



### Procedure: [Procedure Name]

**Alert name:** `[PagerDuty / Prometheus alert name]`
**Trigger:** [What condition fires this alert — e.g., "Error rate > 5% for 5 minutes"]
**Severity:** P1 – Critical | P2 – High | P3 – Medium

#### Symptoms

- Alert: `[alert name]` fires in `[monitoring tool]`
- Users report: [symptom visible to end users]
- Observable in logs: `[log pattern or error message to look for]`
- Observable in metrics: [specific metric and threshold]

#### Investigation Steps

1. **Confirm the alert is not a false positive:**
   ```bash
   # Check current error rate
   kubectl top pods -n [namespace]
   kubectl logs -n [namespace] -l app=[app-label] --tail=100 | grep ERROR
   ```

2. **Identify the affected component:**
   ```bash
   # List pod status
   kubectl get pods -n [namespace] -l app=[app-label]
   # Check recent events
   kubectl describe pods -n [namespace] -l app=[app-label] | tail -50
   ```

3. **Check upstream dependencies:**
   ```bash
   # Verify database connectivity
   kubectl exec -n [namespace] [pod-name] -- pg_isready -h [db-host] -p 5432
   # Check Kafka consumer lag
   kafka-consumer-groups.sh --bootstrap-server [broker] --describe --group [group-id]
   ```

4. **Review recent deployments:**
   - Check deployment history: `kubectl rollout history deployment/[name] -n [namespace]`
   - Compare with last known-good deployment time

#### Remediation Steps

**Option A: Restart affected pods (low risk)**
```bash
# Restart the deployment (triggers rolling restart)
kubectl rollout restart deployment/[deployment-name] -n [namespace]
# Watch rollout status
kubectl rollout status deployment/[deployment-name] -n [namespace]
```
**Expected outcome:** Pods restart within 2–3 minutes. Error rate should drop within 5 minutes.
**When to use:** When logs show memory leak, connection pool exhaustion, or hung goroutines.

**Option B: Roll back to previous version (moderate risk)**
```bash
# Roll back to previous deployment
kubectl rollout undo deployment/[deployment-name] -n [namespace]
# Verify rollback
kubectl rollout status deployment/[deployment-name] -n [namespace]
```
**Expected outcome:** Previous version deployed within 3–5 minutes.
**When to use:** When issue started after a deployment and root cause is unknown.

**Option C: Scale up (moderate risk)**
```bash
# Increase replica count to absorb traffic spike
kubectl scale deployment/[deployment-name] --replicas=[N] -n [namespace]
```
**Expected outcome:** Additional pods available within 60 seconds.
**When to use:** When CPU or memory is saturated due to traffic spike.

#### Verification

- [ ] Error rate returns below 1% on the [dashboard name] dashboard
- [ ] All pods show `Running` status: `kubectl get pods -n [namespace]`
- [ ] Synthetic check passes: [link to synthetic monitor]
- [ ] Confirm with a user-facing smoke test: [describe manual check or test URL]

#### Escalation Criteria

If not resolved within **15 minutes**, escalate to:
- **Engineering on-call lead:** [PagerDuty escalation policy link]
- **Service owner:** [Name] — contact via [Slack handle or phone]
- **DBA (for database issues):** [Name / on-call rotation link]

---

## Escalation Policy

| Level | Who | Contact | When to escalate |
|-------|-----|---------|-----------------|
| L1 | On-call engineer | PagerDuty rotation | Immediately on P1/P2 alert |
| L2 | On-call lead | [PagerDuty escalation] | After 15 min without resolution |
| L3 | Service owner | [Slack / phone] | Customer impact confirmed or 30 min unresolved |
| L4 | Engineering VP | [Phone] | P1 > 30 min or significant revenue impact |

---

## On-Call Contacts

| Role | Name | Slack | Phone |
|------|------|-------|-------|
| Service owner | [Name] | @[handle] | [number] |
| Primary DBA | [Name] | @[handle] | [number] |
| Security (for breaches) | [Name / team] | #security-oncall | [number] |
| Vendor support | [Vendor] | N/A | [support line + ticket URL] |

---

## Changelog

| Version | Date | Author | Changes |
|---------|------|--------|---------|
| v1.0 | [YYYY-MM-DD] | [Name] | Initial runbook |
| v1.1 | [YYYY-MM-DD] | [Name] | Added Procedure: [name] after incident INC-[NNN] |
| v1.2 | [YYYY-MM-DD] | [Name] | Updated escalation contacts |

Pros and cons

Writing guidance

Write runbook procedures from the perspective of an engineer who has never seen this service before and is paged at 3am with adrenaline running high. Every step should be explicit, include the exact command to run, and explain what the expected output is. "Check the logs" is not a procedure step; "Run kubectl logs -n payments -l app=order-service --tail=200 | grep 'ERROR\|WARN' and look for connection timeout errors from the database layer" is a procedure step.

Each procedure must end with a verification section. Engineers under pressure tend to assume a remediation worked after executing the steps; the verification section prevents premature resolution of incidents. Verification should use a different signal than the one that triggered the alert — if the alert was based on error rate, verify with a synthetic user journey or a direct curl to the service health endpoint, not just by watching the same error rate graph drop.

Link runbooks directly from alerts. The moment an alert fires, the on-call engineer should be one click away from the relevant runbook section. If using PagerDuty, add the runbook URL to the alert definition. If using Prometheus Alertmanager, use the runbook_url annotation on each alert rule. This eliminates the friction of searching for documentation during an incident, which often causes engineers to skip the runbook entirely.

Common mistakes

• Runbook rot: Procedures reference service names, URLs, or credentials that no longer exist, causing on-call engineers to waste time hunting for current information during an incident.
• Missing escalation criteria: Engineers spend too long attempting fixes they are unlikely to resolve instead of escalating; explicit time-based escalation criteria prevent this.
• Procedures without context: Steps that say "restart the service" without explaining why or what the expected outcome is leave engineers unsure if the restart worked and uninformed about the underlying issue.
• No alert-to-runbook linkage: A runbook that exists but is not linked from the alert that triggers it might as well not exist — engineers under stress will not search for it.

Review checklist

• Does every procedure include the specific command(s) to run, not just a description of what to do?
• Does every procedure end with explicit verification steps that confirm resolution?
• Are escalation criteria time-bounded (e.g., "escalate after 15 minutes") rather than vague (e.g., "escalate if needed")?
• Is the runbook linked directly from every alert definition it covers?
• Have access requirements been verified — can the on-call engineer actually execute every step with their current permissions?

Example usage

• Payment service outage: An on-call engineer receives a P1 alert for elevated error rates on the checkout service. The runbook guides them through confirming the scope, identifying a database connection pool exhaustion as the root cause, applying the documented mitigation (increasing the pool limit via a config change), and verifying recovery — resolving the incident in 8 minutes instead of the prior average of 35.
• Certificate rotation: A quarterly TLS certificate rotation runbook guides the operator through the multi-step process of generating CSRs, coordinating with the PKI team, staging certificates in the secrets manager, and orchestrating a rolling restart — a process infrequent enough that no engineer reliably remembers all the steps.
• Data pipeline backfill: A runbook documents how to trigger a manual backfill of the analytics pipeline after a data gap incident, including the exact Spark job parameters, the verification queries to run against the data warehouse, and the Slack message template to post in the data-consumers channel when the backfill is complete.

Related templates

• Incident Postmortem — incidents resolved via a runbook often produce postmortem action items to improve the runbook or eliminate the underlying cause.
• SLO Document — runbooks are the operational complement to SLO documents; the SLO defines what "healthy" means and the runbook describes how to restore health.
• Observability — effective runbooks depend on good observability infrastructure: the right metrics, logs, and dashboards to diagnose issues.

Further reading

• Google SRE Book – Being On-Call (Chapter 11)
• PagerDuty – What is a Runbook?
• Runbooks for the Modern Era – The Cloud Architect (Medium)