Chapter 10: Operations and Maintenance: On-Call, Monitoring, Incident Response

Why this matters

Chapter 1's SDLC diagram ends with "operations/maintenance," but this stage is frequently under-covered relative to the earlier ones despite consuming a huge share of a system's total lifetime — most software spends far more of its existence being operated and maintained than being initially built. This chapter surveys the core operational practices at the level every engineer should understand, even those not primarily focused on operations work.

On-call: shared responsibility for a running system

[BEGINNER] On-call rotations assign responsibility for responding to production issues (typically via a paging tool like PagerDuty) to a rotating set of engineers, ensuring someone is always reachable when something goes wrong, even outside normal working hours. A healthy on-call practice balances real responsiveness needs against sustainable workload — an on-call rotation that pages too frequently, for issues that don't actually need immediate human response, burns out engineers and, worse, trains them to treat pages as less urgent than they actually are (directly the same "cry wolf" dynamic that undermines any warning system used too liberally).

## Reasonable on-call paging criteria
PAGE (wakes someone up, needs response within minutes):
- Service is down/returning errors for a significant share of users.
- A critical business function (checkout, payments) is failing.

DO NOT PAGE (can wait for business hours, or send as a non-urgent notification):
- A metric crossed a soft, informational threshold with no user-facing impact.
- A non-critical background job is running slower than usual but still completing.

Monitoring and alerting: turning system behavior into actionable signal

[BEGINNER] Monitoring collects data about a system's behavior (metrics like request latency and error rate, logs, traces); alerting turns specific monitored conditions into notifications when something needs attention. The key design principle, directly connected to the on-call discussion above: an alert should be actionable — if an engineer receiving a specific alert wouldn't know what to do about it, or if the condition doesn't actually require any action, the alert is noise, not signal, and noise specifically erodes trust in the entire alerting system over time (again, the "cry wolf" dynamic).

Poor alert: "CPU usage is above 70%." (Is this actually a problem? Does
             it require action? Many services run healthily well above
             this without any user impact at all -- this alert provides
             no clear signal about whether action is needed.)

Good alert: "Checkout API error rate exceeded 5% for 3 consecutive
             minutes." (Clear, actionable, directly tied to genuine
             user-facing impact -- an engineer receiving this immediately
             knows both that something is wrong AND roughly where to look.)

Incident response: a structured process for handling things going wrong

[BEGINNER] A structured incident response process, run consistently every time (not improvised fresh under the stress of each individual incident), typically includes:

1. DETECTION: an alert fires, or a person notices/reports an issue.
2. TRIAGE: assess severity (how many users affected, how severe) and
   assign an INCIDENT COMMANDER -- one person coordinating the response,
   even if they aren't the one actually fixing the technical issue.
3. MITIGATION: take the fastest safe action to reduce user impact --
   often a rollback (Chapter 9) or a feature flag toggle, NOT necessarily
   the final, complete fix -- restoring service quickly matters more
   initially than finding the root cause immediately.
4. COMMUNICATION: keep stakeholders informed at a regular cadence, even
   with "no new update yet" -- silence during an incident is worse than
   a repeated "still investigating."
5. RESOLUTION: the underlying issue is actually fixed (not just mitigated).
6. POSTMORTEM: a blameless review of what happened, why, and what
   process/system changes will help prevent recurrence (see below).

Blameless postmortems: learning from incidents without punishing individuals

[SENIOR INSIGHT] A blameless postmortem explicitly separates "what happened and why" from "whose fault was it" — not because individual actions don't matter, but because a postmortem culture that assigns blame reliably produces worse future outcomes: engineers become incentivized to hide mistakes, avoid volunteering information that might implicate them, and generally optimize for not looking bad rather than for the team's genuine collective understanding of what actually happened and how to prevent it. A blameless postmortem instead treats an incident as a symptom of a system (technical and process) that allowed the mistake to happen and, ideally, should be redesigned so a similar mistake becomes harder to make in the future — "why did our deployment process allow an unreviewed config change to reach production" rather than "why did this specific engineer make this specific mistake."

## Postmortem: Checkout Outage, 2026-01-15

### What happened
A configuration change intended for the staging environment was
accidentally applied to production, causing checkout to fail for 40
minutes.

### Why (contributing factors, not blame)
- Our deployment tooling doesn't visually distinguish staging vs.
  production targets clearly enough in its UI.
- No approval step exists for production config changes, unlike code
  deployments which require review.

### What we're changing
- Add a distinct visual warning (and a confirmation step) for any
  production-targeted config change.
- Require the same review process for production config changes as
  for code deployments.

Common mistakes

1. Paging on-call engineers for non-actionable or non-urgent conditions, training the team to treat pages as less urgent than they actually are, and contributing to burnout.
2. Creating alerts that aren't clearly actionable, producing noise that erodes trust in the alerting system over time — every alert should map to a clear "if this fires, do X" response.
3. Skipping incident communication because "there's no update yet," leaving stakeholders in the dark — silence during an incident is worse than a repeated, honest "still investigating."
4. Running postmortems that assign individual blame, producing a culture where engineers hide mistakes and avoid volunteering information — directly undermining the team's actual collective learning from the incident.

Checklist: healthy operations and maintenance practices

1. Define clear, actionable paging criteria — page for genuine, urgent, user-impacting issues; use non-urgent notification channels for everything else.
2. Ensure every alert is actionable — if an engineer wouldn't know what to do upon receiving it, or if it doesn't reflect genuine impact, reconsider or remove it.
3. Run a consistent, structured incident response process (detection, triage, mitigation, communication, resolution, postmortem) rather than improvising fresh under stress each time.
4. Conduct blameless postmortems focused on systemic contributing factors, not individual fault — and ensure they produce concrete, tracked follow-up changes, not just a written account of what happened.

Discussion / interview questions

1. "What makes a good, actionable alert, versus a poor one?" Strong answer: A good alert maps to a clear response an engineer would know to take, and reflects genuine, meaningful impact — a poor alert (like a generic CPU threshold with no clear action or impact tied to it) produces noise that erodes trust in the alerting system over time.

2. "Why is a blameless postmortem culture generally more effective than one that assigns individual blame?" Strong answer: Blame-assigning cultures incentivize hiding mistakes and withholding information that might implicate someone, undermining the team's actual collective understanding of what happened; blameless postmortems focus on systemic contributing factors, producing genuine process/system improvements that make the same mistake harder for anyone to make in the future.

3. "During an incident, is it better to find and fix the root cause immediately, or to mitigate user impact first?" Strong answer: Mitigate first — the fastest safe action to reduce user impact (often a rollback or feature flag toggle) matters more initially than finding and fixing the root cause, which can happen after service is restored; prioritizing a slower, complete fix over fast mitigation needlessly extends user impact.

4. "How would you decide what should page an on-call engineer versus what should just generate a non-urgent notification?" Strong answer: Page for conditions with genuine, urgent, user-impacting severity requiring response within minutes; use non-urgent channels for informational thresholds or issues that can reasonably wait for business hours — over-paging trains engineers to under-react to pages and contributes to burnout.

Chapter summary / cheat sheet

• On-call balances responsiveness against sustainable workload — page only for genuinely urgent, actionable, user-impacting issues.
• Monitoring/alerting: every alert should be actionable and tied to genuine impact; non-actionable alerts are noise that erodes trust in the system.
• Incident response: a consistent structured process (detect, triage, mitigate, communicate, resolve, postmortem) — mitigate user impact fast, find root cause after.
• Blameless postmortems focus on systemic contributing factors, not individual fault — this produces better collective learning and genuine process improvements, while blame-assigning cultures incentivize hiding mistakes.

---