Chapter 8: Documentation Across the Lifecycle: Specs, ADRs, Runbooks

Why this matters

Documentation is frequently treated as a nice-to-have that gets cut under deadline pressure — and this is a mistake specifically because different kinds of documentation serve genuinely different, non-overlapping purposes across a project's lifecycle, several of which have no substitute once the knowledge they'd capture has left the team (through memory fading or people leaving). This chapter covers the three documentation types worth deliberately maintaining, and why each earns its place.

Specs: capturing intended behavior before and during implementation

[BEGINNER] A specification (spec) captures what a system or feature is intended to do — directly related to Chapter 4's requirements and Chapter 5's design output, but specifically framed as the durable, referenceable artifact that outlives the initial conversation that produced it. A spec's value isn't just at the moment it's written — it's the reference point for resolving future ambiguity ("was this behavior intentional, or a bug?") and for onboarding someone unfamiliar with the feature's intended design.

Architecture Decision Records (ADRs): capturing why, not just what

[SENIOR INSIGHT] An ADR is a short, focused document capturing one specific architectural or technical decision: the context that prompted it, the options considered, the choice made, and the reasoning — directly the companion Coding Standards book's Chapter 4 "explain why, not what" principle, applied to architectural decisions specifically, at the scale where its absence causes the most damage.

# ADR-012: Use PostgreSQL instead of MongoDB for the orders service

## Status: Accepted

## Context
The orders service needs to store order data with strong consistency
requirements (Database guide, Chapter 3) -- specifically, inventory
decrements must be atomic with order creation to prevent overselling.

## Options considered
1. MongoDB -- team has some existing familiarity; flexible schema.
2. PostgreSQL -- strong ACID transaction support; less schema flexibility.

## Decision
PostgreSQL, specifically because of the atomicity requirement for
inventory decrements (see Database guide, Chapter 3's oversell scenario)
-- MongoDB's weaker default transactional guarantees at the time of this
decision made it a worse fit for this specific requirement, despite the
team's greater familiarity with it.

## Consequences
Team needs to invest in PostgreSQL operational familiarity. Revisit if
inventory management requirements change significantly.

[IN PRACTICE] The specific, concrete payoff of an ADR shows up months or years later, when someone (possibly the original decision-maker, who has simply forgotten the details) asks "why don't we just use MongoDB here, like the rest of our systems?" — without an ADR, this question either goes unanswered (risking a costly, unnecessary re-litigation or an accidental reversal of a decision made for good reasons that have been forgotten) or requires archaeology through old chat logs and half-remembered conversations. With an ADR, the answer is one document away, complete with the actual reasoning, not just the conclusion.

Runbooks: capturing operational knowledge for when things go wrong

[BEGINNER] A runbook documents the specific steps to diagnose and resolve a known operational issue — written before an incident, specifically so that whoever is on-call during an actual incident (who may not be the engineer most familiar with the system, especially at 3 AM) has a concrete, followable procedure rather than needing to reconstruct expert knowledge under real production pressure.

# Runbook: Order Processing Queue Backing Up

## Symptoms
- PagerDuty alert: "order_queue_depth > 1000"
- Orders are being accepted but not processed within the normal ~30s window.

## Diagnosis steps
1. Check queue consumer health: `kubectl get pods -l app=order-processor`
   -- are consumer pods actually running and not crash-looping?
2. Check for a recent deploy: has `order-processor` been deployed in the
   last hour? A recent deploy is the most common cause.
3. Check downstream dependency health (payment gateway status page) --
   a downstream outage can cause processing to stall even with healthy consumers.

## Resolution
- If consumers are crash-looping: check logs for the crash reason, likely
  rollback candidate: `kubectl rollout undo deployment/order-processor`
- If a downstream dependency is down: this is expected backpressure
  (see companion Networking guide, Chapter 11) -- monitor for the queue
  to drain once the dependency recovers; escalate to the payment team if
  down for more than 15 minutes.

## Escalation
If unresolved after 20 minutes, page @orders-team-lead.

[SENIOR INSIGHT] A runbook's real value is inversely related to how familiar the on-call engineer is with the system — for the system's original author, a runbook might feel redundant ("I already know all this"); for anyone else on a rotation, it's the difference between a confident, fast resolution and a stressful, slow one under real production pressure, directly serving the companion Code Reviews book's Chapter 1 bus-factor/shared-ownership theme, applied to operational knowledge specifically rather than code understanding.

Common mistakes

1. Treating all documentation as one undifferentiated category to cut under deadline pressure, missing that specs, ADRs, and runbooks serve genuinely different purposes, some of which (especially ADRs and runbooks) have no substitute once the knowledge fades from memory or the people who had it leave.
2. Writing an ADR that only records the decision, not the reasoning and alternatives considered — this provides much less future value than a full ADR, since it can't answer "why not the alternative" when that question inevitably comes up later.
3. Writing a runbook only after an incident has already happened once, rather than proactively for known, anticipatable operational risks — though writing one immediately after a first occurrence (so at least the second occurrence is well-handled) is still far better than never writing one at all.
4. Letting documentation drift out of sync with reality without any process to catch it — an out-of-date runbook that no longer matches the actual system can be actively dangerous during a real incident, actively worse than having no runbook and thus no false confidence.

Checklist: maintaining useful documentation across the lifecycle

1. Write a spec capturing intended behavior for anything beyond a trivial feature, serving as a durable reference point beyond the initial conversation that produced it.
2. Write an ADR for any significant architectural or technology decision, capturing context, alternatives considered, the decision, and the reasoning — not just the conclusion.
3. Write runbooks proactively for known, anticipatable operational risks, and reactively (immediately) after any new incident type occurs for the first time.
4. Periodically review and update documentation (especially runbooks, which need to stay accurate to remain safe to rely on during a real incident) rather than letting it silently drift out of sync with the actual system.

Discussion / interview questions

1. "What's the difference between a spec, an ADR, and a runbook, and when would you write each?" Strong answer: A spec captures intended feature/system behavior (requirements and design); an ADR captures a specific architectural decision's context, alternatives, and reasoning; a runbook captures operational diagnosis/resolution steps for a known issue — each serves a distinct lifecycle purpose and audience.

2. "Why is an ADR that only records 'we chose X' less valuable than one that also records the alternatives considered and the reasoning?" Strong answer: Without the alternatives and reasoning, the ADR can't answer "why not Y" when that question inevitably comes up later — the full context is what prevents costly re-litigation or accidental reversal of a decision made for reasons that have since been forgotten.

3. "Why might a runbook be more valuable to an on-call engineer than to the system's original author?" Strong answer: The original author already has the expert knowledge a runbook encodes; for anyone else on a rotation, especially under real production pressure, the runbook is the difference between a confident, fast resolution and a slow, stressful one — directly serving the bus-factor/shared-ownership value of documented operational knowledge.

4. "What's more dangerous: having no runbook for an operational issue, or having an out-of-date one?" Strong answer: An out-of-date runbook can be more dangerous, since it provides false confidence — an on-call engineer following stale steps during a real incident might take an action that no longer matches the actual system's current behavior, whereas no runbook at least prompts appropriate caution and first-principles investigation.

Chapter summary / cheat sheet

• Specs capture intended behavior as a durable reference beyond the initial conversation; ADRs capture the reasoning behind a specific architectural decision (context, alternatives, choice, consequences); runbooks capture operational diagnosis/resolution steps for known issues.
• An ADR's value is in the reasoning and alternatives considered, not just the final decision — this is what prevents costly re-litigation or accidental reversal later.
• Runbooks provide the most value to whoever is on-call and least familiar with a system — directly serving shared-ownership/bus-factor goals for operational knowledge.
• Documentation needs periodic review to stay accurate — an out-of-date runbook can be actively more dangerous than no runbook at all.

---