Chapter 8: Observability: Metrics, Logging, Distributed Tracing, and Actionable Alerts

Why this matters

The companion SDLC book's Chapter 10 introduced monitoring and alerting at a survey level, with actionable alerting as its core lesson. This chapter goes deeper into the three specific observability signal types — metrics, logs, and distributed traces — and why a mature system needs all three, since each answers a genuinely different diagnostic question that the others can't.

The three pillars, and what each one answers

[BEGINNER]

• Metrics — numeric measurements aggregated over time (request rate, error rate, latency percentiles, CPU usage). Answers: "what is the overall health/behavior trend of this system right now, or over time?" Cheap to store and query even at very high volume, since they're pre-aggregated.
• Logs — discrete, timestamped records of specific events ("user 42 login failed: invalid password"). Answers: "what exactly happened, in detail, for this specific event or request?" More expensive to store at volume, but essential for understanding a specific incident's exact sequence of events.
• Distributed tracing — follows a single request's entire journey across multiple services, recording how long each step took. Answers: "for this one specific slow/failed request, exactly which service or step in its journey was responsible?"

Metrics tell you: "checkout latency has degraded across the board over
                   the last 10 minutes."

Logs tell you: "this SPECIFIC checkout request at 14:32:07 failed with
                a database connection timeout error."

Distributed tracing tells you: "for this SPECIFIC slow checkout request,
                                 380ms of its total 420ms was spent
                                 waiting on the Inventory Service
                                 specifically, not the Payment Service
                                 or the Database."

Why a real incident usually needs all three, in sequence

[SENIOR INSIGHT] These three signals aren't redundant alternatives — a real incident investigation typically moves through all three, in a specific, natural order: metrics first reveal that something is wrong and roughly when it started (a dashboard showing latency or error rate spiking); distributed tracing narrows down where in a multi-service request path the problem is actually occurring; logs for that specific service, around that specific time, reveal exactly what went wrong at a detailed, mechanistic level. Skipping straight to logs without metrics or tracing to narrow the search first means searching through an enormous volume of largely-irrelevant log entries across every service, without knowing which service or time window actually matters — directly analogous to the companion Database book's Chapter 14 diagnostic sequence (measure with EXPLAIN ANALYZE before guessing at a fix).

Effective incident diagnosis sequence:
1. METRICS dashboard shows checkout latency spiking at 14:30.
2. DISTRIBUTED TRACING for slow checkout requests around 14:30 shows
   the Inventory Service specifically is the bottleneck.
3. LOGS for the Inventory Service around 14:30 reveal it was
   repeatedly retrying a failed database connection.
-- each signal narrowed the search space for the next one, rather
   than searching blindly through logs from every service at once.

Structured logging: making logs actually queryable at scale

[IN PRACTICE] Unstructured, free-text log messages ("User login failed for user 42") are readable by a human scanning a small number of them, but become nearly unsearchable at real production log volume. Structured logging (emitting logs as machine-parseable key-value data, typically JSON) makes logs genuinely queryable and filterable at scale:

{"timestamp": "2026-01-15T14:32:07Z", "level": "error", "event": "login_failed",
 "user_id": "42", "reason": "invalid_password", "service": "auth-service"}

This structure lets an engineer query "show me all login_failed events for user_id=42 across the last hour" directly and precisely, rather than grep-ing through free-text log lines and hoping the search terms happen to match how a specific message was phrased.

Actionable alerting, restated with this chapter's added precision

[BEGINNER] Directly the companion SDLC book's Chapter 10 principle, refined with this chapter's vocabulary: an alert should typically fire on a metric (a fast, cheap, pre-aggregated signal) reflecting genuine, meaningful user impact — not on a raw log event (too noisy, too voluminous, too easy to false-positive on a single, non-representative occurrence) and not requiring a full distributed trace to evaluate (too slow and expensive to compute for every potential alert check).

Common mistakes

1. Relying on only one of the three observability signal types, missing that each answers a genuinely different diagnostic question the others structurally can't — a system with rich metrics but no distributed tracing can tell you that and roughly when something's wrong, but not efficiently where in a multi-service path.
2. Searching through raw logs first, without using metrics or tracing to narrow the search space, wasting significant time scanning largely-irrelevant log volume across services and time windows that don't actually matter for the specific incident.
3. Using unstructured, free-text logging at real production scale, making logs effectively unsearchable and forcing engineers to guess at exact phrasing when trying to query for a specific event type.
4. Alerting directly on raw log events or expensive per-check distributed traces, rather than on fast, pre-aggregated metrics reflecting genuine, meaningful impact.

Discussion / interview questions

1. "What are the three pillars of observability, and what distinct question does each answer?" Strong answer: Metrics answer "what's the overall health/trend right now" (aggregated, cheap); logs answer "what exactly happened for this specific event" (detailed, more expensive at volume); distributed tracing answers "where in a multi-service request path did the problem occur" (per-request, cross-service).

2. "Why would you typically use metrics, then tracing, then logs, in that order, during an incident investigation?" Strong answer: Metrics reveal that something is wrong and roughly when; tracing narrows down which specific service/step in a request path is responsible; logs for that specific service and time window then reveal the exact, detailed cause — each step narrows the search space for the next, avoiding an unfocused search through all logs from all services at once.

3. "What is structured logging, and why does it matter at real production scale?" Strong answer: Emitting logs as machine-parseable key-value data (typically JSON) rather than free text — this makes logs genuinely queryable and filterable by specific fields at scale, rather than requiring engineers to grep through free-text messages and guess at exact phrasing.

4. "Why should alerts typically be based on metrics rather than raw log events or full distributed traces?" Strong answer: Metrics are fast, cheap, and pre-aggregated, making them suitable for continuous, real-time evaluation; raw log events are too noisy and volume-heavy for reliable alerting, and computing a full distributed trace for every potential alert check would be far too slow and expensive to run continuously.

Chapter summary / cheat sheet

• Metrics (aggregated trend, cheap), logs (detailed per-event record, more expensive at volume), and distributed tracing (per-request, cross-service journey) each answer a genuinely distinct diagnostic question.
• A real incident investigation typically moves through all three in sequence: metrics reveal that/when, tracing reveals where, logs reveal exactly what — each step narrowing the search space for the next.
• Structured logging (JSON, key-value) makes logs genuinely queryable at production scale, unlike free-text messages that become unsearchable at volume.
• Alerts should be based on fast, cheap, pre-aggregated metrics reflecting genuine impact — not raw log events or expensive per-check distributed traces.

---