Incident Postmortem Template

What it is

An incident postmortem (also called a post-incident review, PIR, or retrospective) is a structured document that captures what happened during a significant incident, why it happened, what the impact was, and what the team will do to prevent recurrence. It is written after the incident is resolved, typically within 24 to 72 hours while details are fresh, and reviewed by the team in a collaborative meeting.

The defining principle of the modern postmortem is blamelessness. The goal is not to identify who made a mistake but to understand the systemic conditions, tooling gaps, and process failures that made a human error possible or that amplified its impact. As Google's SRE culture famously articulates: if a single engineer's action can cause a major outage, the system is at fault, not the engineer — and the system needs to be made safer.

A well-written postmortem is one of the highest-leverage documents an engineering team can produce. It transforms a painful experience into organizational learning, generates concrete action items that improve reliability, and builds psychological safety by demonstrating that incidents are treated as learning opportunities rather than accountability events.

Why it exists

Without postmortems, the same incidents tend to recur. The engineer who resolved last month's outage carries the root cause understanding in their head, but that knowledge does not transfer to the rest of the team. The contributing systemic issues — the missing alert, the inadequate runbook, the lack of circuit breaking — remain in place, ready to cause the next incident. Postmortems make the implicit explicit, forcing the team to translate tacit knowledge into documented improvements.

Postmortems also close the feedback loop between incidents and system design. When action items from postmortems are tracked, prioritized, and completed, engineering teams can observe a measurable decrease in incident frequency and severity over time. Organizations that skip postmortems or write them without tracking follow-through tend to experience the same class of incidents repeatedly, eroding customer trust and team morale.

For organizations subject to compliance requirements (SOC 2, ISO 27001, PCI-DSS), postmortem documentation provides evidence of a functioning incident management process — an auditable record that incidents are investigated, root causes identified, and corrective actions taken. Beyond compliance, postmortem archives are invaluable for technical due diligence and for building credibility with enterprise customers who ask about your reliability practices.

When to use

• After any P1 or P2 incident that caused user-facing impact, data loss, or a security event, regardless of duration.
• After any incident that significantly exceeded its expected MTTR, even if the final impact was limited.
• After near-misses — events that did not cause user impact but revealed a serious vulnerability in the system or process.
• After incidents that required escalation beyond the initial on-call responder, indicating the runbook was insufficient.
• When a pattern of minor incidents suggests an underlying systemic issue worth analyzing holistically rather than incident by incident.

When NOT to use

• For minor, quickly-resolved incidents with no user impact and a clear, simple cause — a brief incident ticket update is sufficient.
• As a punitive mechanism — if postmortems are used to assign blame or document who made the mistake, they will be written defensively and the team will lose the psychological safety needed for honest root cause analysis.
• As a substitute for action — writing a thorough postmortem and then not completing the action items is worse than not writing one, as it signals that the process is theater.
• So frequently that the team cannot keep up — postmortem fatigue is real. For high-frequency minor incidents, consider a weekly batch review rather than individual postmortems for each event.

Template

# Incident Postmortem: [Brief Incident Title]

**Incident ID:** INC-[NUMBER]
**Date of Incident:** [YYYY-MM-DD]
**Date of Review Meeting:** [YYYY-MM-DD]
**Severity:** P1 – Critical | P2 – High | P3 – Medium
**Status:** Draft | In Review | Final
**Incident Commander:** [Name]
**Document Owner:** [Name — person responsible for keeping this doc up to date]
**Participants:** [Name (role)], [Name (role)], [Name (role)]

---

## Incident Summary

On [date], [brief description of what failed] caused [impact on users/business]
for approximately [duration]. The root cause was [one-sentence root cause].
The incident was resolved at [time] by [brief description of resolution].
[N] action items have been identified to prevent recurrence.

**Total incident duration:** [X hours Y minutes]
**Time to detect:** [X minutes from first impact to alert/detection]
**Time to mitigate:** [X minutes from detection to service restoration]
**Affected services:** [List of services]

---

## Timeline


| Time (UTC) | Event |
|------------|-------|
| HH:MM | [Deployment / config change / external trigger] |
| HH:MM | [First anomalous metric observed — e.g., "Error rate begins rising"] |
| HH:MM | [Alert fires / on-call paged] |
| HH:MM | [Incident commander declares incident, opens bridge] |
| HH:MM | [Investigation action taken — e.g., "Reviewed recent deployments"] |
| HH:MM | [Hypothesis formed — e.g., "Suspected database connection issue"] |
| HH:MM | [Remediation attempt 1 — e.g., "Restarted worker pods"] |
| HH:MM | [Outcome of attempt — e.g., "No improvement observed"] |
| HH:MM | [Remediation attempt 2 — describe action] |
| HH:MM | [Service restored — error rate returns to baseline] |
| HH:MM | [Incident resolved and all-clear communicated] |
| HH:MM | [Post-incident monitoring period ends, incident closed] |

---

## Root Cause Analysis


### Immediate Trigger
[The direct cause of the incident — the specific event that initiated the failure.]

### Root Cause
[The underlying systemic reason the trigger was possible and/or had the impact it did.
Example: "The deployment pipeline did not validate the config schema before deploying,
allowing a malformed value to reach production. No automated test covered this
configuration path."]

### Contributing Factors

1. **[Factor]:** [Description — e.g., "Alert threshold was set too high, delaying detection by 18 minutes"]
2. **[Factor]:** [Description — e.g., "Runbook did not cover this failure mode, adding 15 minutes to MTTR"]
3. **[Factor]:** [Description — e.g., "No circuit breaker between Service A and Service B; failure cascaded"]
4. **[Factor]:** [Description — e.g., "Database connection pool limit not documented, responder unaware of tunable"]

---

## Impact Assessment

### User Impact
- **Users affected:** [Number or percentage]
- **Requests failed:** [Count or percentage during incident window]
- **Features unavailable:** [List of specific features or workflows that were broken]
- **Geographical scope:** [Global / specific regions]

### Business Impact
- **Revenue impact:** [Estimated $X (if quantifiable) or "under investigation"]
- **SLO impact:** [X minutes of error budget consumed; Y% of monthly budget remaining]
- **Customer communications sent:** [Yes/No — link to status page update or customer email]
- **Regulatory / compliance implications:** [Any data breach, PII exposure, or compliance event]

---

## Action Items


| # | Action | Type | Owner | Due Date | Ticket |
|---|--------|------|-------|----------|--------|
| 1 | [Description of action] | Prevention | [@name] | [YYYY-MM-DD] | [TICKET-NNN] |
| 2 | [Description of action] | Detection | [@name] | [YYYY-MM-DD] | [TICKET-NNN] |
| 3 | [Description of action] | Mitigation | [@name] | [YYYY-MM-DD] | [TICKET-NNN] |
| 4 | [Description of action] | Process | [@name] | [YYYY-MM-DD] | [TICKET-NNN] |

**Action item types:**
- **Prevention:** Prevents the root cause from occurring again
- **Detection:** Makes future occurrences of this class of issue faster to detect
- **Mitigation:** Reduces the impact or MTTR when a similar incident occurs
- **Process:** Improves the incident response process itself

---

## Lessons Learned


### What went well
* [Something the team did effectively during the incident response]
* [A process or tool that worked as intended and helped]
* [An individual action or decision that reduced impact]

### What did not go well
* [Something that slowed detection, diagnosis, or resolution]
* [A gap in tooling, documentation, or process that was exposed]
* [A communication breakdown or decision that had negative consequences]

### Where we got lucky

* [A factor that prevented this from being a worse incident than it was]
* [A manual check or human intervention that caught a secondary issue]

---

## Appendix

### Raw Metrics
[Link to dashboard snapshot or embed key charts showing the incident window]

### Relevant Log Excerpts
```
[Paste key log lines that illustrate the failure, with timestamps and context]
```

### Slack / Communication Thread
[Link to the incident Slack thread or bridge recording]

### Related Incidents
- [INC-NNN — similar past incident, link to postmortem if it exists]

Pros and cons

Writing guidance

Write the timeline first. Pull raw data from PagerDuty, Slack, deployment logs, and monitoring dashboards before memories fade. The timeline should be a factual sequence of events with precise timestamps — not an interpretation. Separate "what happened" from "why it happened." Engineers often conflate these, which leads to timelines contaminated with hindsight analysis that obscures the actual sequence of events as they were experienced.

The root cause section requires the discipline to ask "why" enough times. Most immediate causes are themselves symptoms of a deeper systemic issue. "The deployment included a bug" is not a root cause. "The deployment pipeline did not run integration tests against the staging environment before promoting to production, because the test suite was too slow and was disabled six months ago without being re-enabled" is a root cause. The deeper you go, the more durable your action items will be.

Action items are where postmortems create value. Each action item should be specific enough that its completion can be verified: "improve monitoring" is not an action item; "add an alert that fires when the database connection pool utilization exceeds 80% for two consecutive minutes" is. Assign a single named owner per item — shared ownership is no ownership. Set a deadline and link to a tracking ticket before the review meeting ends.

Common mistakes

• Writing a timeline without precise timestamps: "In the morning" or "after the alert" are useless for analysis. Every timeline event needs a timestamp to the minute, in a consistent timezone.
• Action items owned by teams instead of individuals: "The SRE team will improve alerting" will not get done. "@alice will add alert TICKET-123 by 2025-02-15" will.
• Skipping the "where we got lucky" section: Near-misses and lucky breaks are among the highest-signal inputs for improving reliability. A system where a single manual intervention prevented data loss should be automated or hardened.
• Never publishing or reviewing past postmortems: Postmortem value compounds when the organization reads and references past incidents. Build a searchable postmortem archive and reference relevant past incidents in new postmortems.

Review checklist

• Does the timeline use precise UTC timestamps pulled from actual system logs rather than recalled estimates?
• Does the root cause section go beyond the immediate trigger to identify systemic contributing factors?
• Does every action item have a single named owner, a specific due date, and a linked tracking ticket?
• Is the document written blamelessely — are failure descriptions focused on systems and processes rather than individual engineers?
• Has the SLO impact been quantified and the error budget consumption recorded?

Example usage

• Database failover incident: A 47-minute checkout outage caused by an unexpected primary database failover produces a postmortem that identifies three action items: automating connection string updates via service discovery, adding a circuit breaker to prevent cascading failure to the API layer, and updating the runbook to include the database failover procedure.
• Deployment-caused regression: A bad deployment that degraded search functionality for 2 hours produces action items to add canary deployment analysis to the pipeline and to set up automated rollback triggers based on error rate thresholds.
• Security near-miss: A misconfigured S3 bucket that was publicly readable for 6 hours but contained no sensitive data triggers a postmortem that produces an infrastructure-as-code policy to block public bucket ACLs and a weekly automated scan for public bucket configurations.

Related templates

• Runbook — postmortem action items frequently produce runbook improvements; the runbook is the operational counterpart to the postmortem.
• SLO Document — postmortems should quantify error budget consumption; the SLO document defines the budget being consumed.
• Reliability — postmortems feed directly into reliability engineering practices and system hardening work.

Further reading

• Google SRE Book – Postmortem Culture: Learning from Failure (Chapter 15)
• Atlassian – Postmortem template and best practices
• Awesome SRE – curated postmortem resources and real-world examples