Chapter 9: Incident Response: Severity, Incident Command, Communication, Blameless Postmortems

Why this matters

The companion SDLC book's Chapter 10 introduced the incident response process and blameless postmortems at a survey level. This chapter goes deeper on two specific pieces that determine whether a structured incident response process actually functions well under real pressure: precisely-defined severity levels, and the incident commander role's specific, non-negotiable responsibilities.

Severity levels: precise definitions, not vague impressions

[BEGINNER] A severity level system only works if its levels are defined precisely enough that different engineers, under real pressure, consistently classify the same situation the same way — a vague definition ("SEV1 = really bad") produces inconsistent classification that undermines the entire point of having levels at all (routing the right level of urgency and resources to a given incident).

## Example severity level definitions
SEV1 (Critical): Complete outage of a core, revenue-critical function
   (checkout, login) affecting ALL or nearly all users. Page immediately,
   all-hands response, executive notification.

SEV2 (Major): Significant degradation or partial outage of a core function
   (checkout works but is very slow, OR affects a significant subset of
   users/one region). Page the on-call team; broader notification if not
   resolved within 30 minutes.

SEV3 (Minor): A non-critical feature degraded or unavailable, workaround
   exists, minimal user impact. Handled during business hours, tracked
   but not paged.

[SENIOR INSIGHT] The precision matters specifically because severity level determines resource allocation and urgency automatically and consistently — a SEV1 justifies pulling multiple engineers off other work immediately and notifying executives; misclassifying a SEV1 as a SEV3 (or the reverse) either under-responds to a genuine crisis or over-mobilizes the organization for something that didn't warrant it, and either error erodes trust in the severity system for future incidents.

The incident commander role: coordination, not necessarily fixing

[SENIOR INSIGHT] A common, understandable but harmful instinct during a real incident: the most senior or most knowledgeable engineer available starts simultaneously trying to diagnose the technical problem and coordinate the response (communication, tracking who's doing what, deciding when to escalate further) — these are genuinely different jobs, and trying to do both simultaneously under real pressure means doing both worse than doing either one well. The incident commander (IC) role exists specifically to separate them: the IC's job is coordination — tracking what's been tried, ensuring communication happens on a regular cadence (SDLC book, Chapter 10), deciding when to escalate or bring in additional help, and making the final call on trade-off decisions (like whether to attempt a risky fix or wait for more certainty) — the IC does not need to be, and often explicitly should not be, the engineer actually debugging the technical problem.

## Incident commander responsibilities (NOT technical debugging)
- Maintain a clear, current understanding of the incident's status and
  who is doing what.
- Ensure stakeholder communication happens on a consistent cadence.
- Make explicit trade-off decisions (proceed with a risky fix now, or
  wait for a safer option?) rather than leaving them ambiguous.
- Decide when to escalate (bring in more engineers, notify leadership).
- Declare the incident resolved, and initiate the postmortem process.

[IN PRACTICE] Separating these roles explicitly means the engineer with the deepest technical context can stay fully focused on diagnosis and fixing, while the IC handles everything else that would otherwise fragment that engineer's attention — this separation becomes more valuable, not less, the more severe and high-pressure an incident is, which is exactly when attention fragmentation is most costly.

Communication cadence: the specific discipline of "no update" updates

[IN PRACTICE] Directly extending the SDLC book's Chapter 10 point that silence during an incident is worse than repeated "still investigating": a mature incident process defines an explicit communication cadence upfront (e.g., "an update every 15 minutes for a SEV1, even if the update is just 'still investigating, no new information'") — this removes the awkward, unstated judgment call of "do I have enough news to justify an update yet," which under real pressure tends to produce longer and longer silences precisely when stakeholders are most anxious for information.

Blameless postmortems, restated with the "systemic factors" discipline made concrete

[SENIOR INSIGHT] Building on the SDLC book's Chapter 10 blameless postmortem principle: a genuinely useful postmortem asks "why did our systems and processes allow this to happen" repeatedly (a version of Chapter 4's five-whys from the SDLC book, applied to incident root-causing specifically) until it reaches a systemic, addressable factor — not stopping at the first, most proximate cause, which is often a specific individual's specific action:

Proximate cause (stop here = blame, unhelpful): "Engineer X pushed a
config change that caused the outage."

Deeper, systemic cause (keep asking why): "Why could a config change
reach production without review?" -> "Our config deployment pipeline
doesn't require the same review as code deployments." -> "Why not?"
-> "We never extended our code review requirements to configuration
changes when we built this pipeline."

Actionable, systemic fix: require the same review process for config
changes as for code changes (directly the SDLC book's Chapter 10 example).

Common mistakes

1. Defining severity levels vaguely, producing inconsistent classification across different engineers and different incidents, undermining the entire point of having a severity system.
2. Having the most technically knowledgeable engineer also serve as incident commander, fragmenting their attention between diagnosis and coordination when a real incident specifically needs full focus on the former.
3. Leaving communication cadence as an ad hoc, unstated judgment call, producing longer silences under pressure precisely when stakeholders most need regular updates.
4. Stopping postmortem analysis at the first, most proximate cause (a specific individual's specific action) rather than continuing to ask "why" until reaching a systemic, addressable factor.

Discussion / interview questions

1. "Why do severity level definitions need to be precise rather than left to individual judgment?" Strong answer: Precise definitions produce consistent classification across different engineers and incidents, which is what lets severity level reliably drive automatic resource allocation and urgency — vague definitions lead to misclassification (under- or over-responding), which erodes trust in the severity system over time.

2. "Why shouldn't the incident commander necessarily be the engineer actually debugging the technical problem?" Strong answer: Diagnosis/fixing and coordination (communication, tracking status, escalation decisions) are genuinely different jobs; combining them in one person under real incident pressure means doing both worse — separating them lets the technical expert stay fully focused on the fix while the IC handles everything else.

3. "Why would a mature incident process define an explicit communication cadence, including sending updates with 'no new information'?" Strong answer: Without an explicit cadence, "do I have enough news to update yet" becomes an ad hoc judgment call that tends to produce longer silences under pressure — exactly when stakeholders are most anxious for information; a defined cadence removes that judgment call entirely.

4. "Why should a postmortem keep asking 'why' past the first, most proximate cause of an incident?" Strong answer: The first cause is often a specific individual's specific action, which produces blame rather than a systemic, addressable fix; continuing to ask why (a version of the five-whys technique) usually reaches a systemic factor (a missing process safeguard) that, once fixed, prevents not just this specific incident but an entire class of similar future ones.

Chapter summary / cheat sheet

• Severity levels need precise, specific definitions to produce consistent classification and reliably drive appropriate, automatic resource allocation.
• The incident commander role is coordination (status tracking, communication, escalation, trade-off decisions) — deliberately separate from technical debugging, especially for severe incidents where attention fragmentation is most costly.
• Define an explicit communication cadence upfront, including sending "no new information" updates — this removes an ad hoc judgment call that tends toward unhelpful silence under pressure.
• Blameless postmortems should keep asking "why" past the first, most proximate (often individual-blaming) cause, until reaching a systemic, genuinely addressable factor.

---