Vasu Patel logoVasu Patel
← Built to Last: Reliability & Scaling
All chapters in this track
  1. 01Chapter 1: Why Reliability and Scale Matter
  2. 02Chapter 2: Reliability Vocabulary: SLAs, SLOs, SLIs, and Error Budgets
  3. 03Chapter 3: Redundancy and Fault Tolerance: Eliminating Single Points of Failure
  4. 04Chapter 4: Horizontal vs. Vertical Scaling, and Load Balancing/Auto-Scaling
  5. 05Chapter 5: Caching for Scale: Cache Stampede and Invalidation at Scale
  6. 06Chapter 6: Database Scaling: Replication, Partitioning, Read Replicas, Connection Pooling
  7. 07Chapter 7: Resilience Patterns: Timeouts, Retries, Circuit Breakers, Bulkheads, Load Shedding
  8. 08Chapter 8: Observability: Metrics, Logging, Distributed Tracing, and Actionable Alerts
  9. 09Chapter 9: Incident Response: Severity, Incident Command, Communication, Blameless Postmortems
  10. 10Chapter 10: Chaos Engineering: Proactively Testing for Failure
  11. 11Chapter 11: Capacity Planning and Load Testing
  12. 12Chapter 12: Core Trade-offs: Consistency vs. Availability, Cost vs. Redundancy, Simplicity vs. Resilience
  13. 13Chapter 13: Incident Case Study: The Cascading Failure
  14. 14Chapter 14: Incident Case Study: The Silent Data Corruption
  15. 15Chapter 15: Common Interview Questions About Reliability and Scaling

Chapter 2 of 15

Chapter 2: Reliability Vocabulary: SLAs, SLOs, SLIs, and Error Budgets

Why this matters

These four terms are used constantly in reliability discussions, frequently interchangeably and imprecisely — and the imprecision is costly, because each one plays a genuinely distinct role, and conflating them leads to real confusion about what a team is actually committing to, measuring, and allowed to trade off.

The four terms, precisely

[BEGINNER]

  • SLI (Service Level Indicator) — an actual, measured metric: "the percentage of requests that succeed," "the 99th percentile response latency."
  • SLO (Service Level Objective) — an internal target for an SLI: "99.9% of requests should succeed," "p99 latency should be under 200ms." This is the team's own goal, not a customer-facing promise.
  • SLA (Service Level Agreement) — an external, often contractual commitment to a customer, usually with a stated consequence for missing it (a service credit, a penalty). An SLA is typically set looser than the internal SLO specifically to provide a safety margin — a team wants room to occasionally miss its own internal target without breaching an external commitment.
  • Error budget — the direct, practical consequence of an SLO: if the SLO is 99.9% availability, the error budget is the remaining 0.1% — a concrete, spendable allowance for failure, planned deployments, and risk.
SLI (measured): 99.95% of requests succeeded this month.
SLO (internal target): 99.9% of requests should succeed.
SLA (external commitment): 99.5% uptime guaranteed to customers,
                             with service credits if missed.

Error budget: 0.1% (from the SLO) -- this month, the team "spent"
              0.05% of it on actual failures, leaving 0.05% remaining
              to spend on calculated risks (a risky deployment, an
              experimental feature) before the internal target is at risk.

Why "five nines" and similar numbers matter concretely

[BEGINNER] Availability targets are usually expressed as a percentage of "nines," and the practical difference between them is often not intuitive until converted into actual allowed downtime:

AvailabilityDowntime per yearDowntime per month
99% ("two nines")~3.65 days~7.3 hours
99.9% ("three nines")~8.76 hours~43.8 minutes
99.99% ("four nines")~52.6 minutes~4.4 minutes
99.999% ("five nines")~5.26 minutes~26 seconds

[SENIOR INSIGHT] The jump between adjacent nines is not a small, incremental improvement — it's roughly a 10x reduction in allowed downtime, which typically demands a qualitatively different (not just quantitatively better) architecture: getting from "three nines" to "four nines" often requires genuine redundancy and automated failover (Chapter 3) that "three nines" might not have strictly needed; getting from "four nines" to "five nines" typically requires eliminating essentially every single point of failure, including ones that are individually very rare, plus extremely fast, fully automated failure detection and recovery. This is precisely why an SLO/SLA should never be chosen arbitrarily ("let's promise five nines, it sounds impressive") — each additional nine has a real, escalating engineering and operational cost that must be justified by genuine business need.

The error budget's real purpose: making risk-taking explicit and data-driven

[SENIOR INSIGHT] An error budget's most valuable practical use isn't just tracking compliance — it's giving a team explicit, quantified permission to take calculated risks, and an equally explicit signal to stop when that permission is exhausted:

## Using an error budget in practice
- Error budget healthy (mostly unspent this period): the team can
  ship a riskier deployment, run a chaos engineering experiment
  (Chapter 10), or take on other calculated reliability risk more
  freely, since there's room to absorb an occasional resulting failure.
- Error budget nearly exhausted: the team should shift focus toward
  reliability work and defer risky changes -- not because of a vague
  "let's be careful" instinct, but because of a concrete, measured signal.

[IN PRACTICE] This turns a common, unproductive organizational tension (engineering wanting to ship features quickly vs. wanting the system to be reliable) into a shared, objective, data-driven conversation — the error budget itself, not a subjective argument about priorities, determines when to prioritize velocity versus reliability work, removing much of the political/interpersonal friction from a decision that would otherwise be argued from first principles every single time.

Common mistakes

  1. Using SLA, SLO, and SLI interchangeably, causing real confusion about whether a specific number is an external, contractual commitment or an internal target the team can more flexibly manage.
  2. Setting an SLA equal to (rather than looser than) the internal SLO, leaving no safety margin — any month where the SLO is narrowly missed becomes an automatic SLA breach with real customer/contractual consequences.
  3. Choosing an availability target (99.99% vs. 99.9%) without connecting it to actual business need and the very real, escalating engineering cost of each additional nine.
  4. Not using the error budget as an actual decision-making tool, tracking it passively without letting it inform real velocity-vs-reliability trade-off decisions as intended.

Discussion / interview questions

  1. "What's the difference between an SLI, an SLO, and an SLA?" Strong answer: SLI is the actual measured metric; SLO is the team's internal target for that metric; SLA is an external, often contractual commitment to customers, typically set looser than the internal SLO to provide a safety margin.

  2. "Why does the difference between 99.9% and 99.99% availability matter so much, even though the numbers look similar?" Strong answer: Each additional nine represents roughly a 10x reduction in allowed downtime, typically requiring a qualitatively different architecture (more thorough redundancy, faster automated failover) rather than just incremental improvement — the engineering cost escalates sharply with each additional nine, which is why the target should be chosen based on genuine business need, not arbitrarily.

  3. "What is an error budget, and how would you use it in practice beyond just tracking compliance?" Strong answer: The allowed amount of unreliability implied by an SLO (100% minus the SLO percentage); used practically as an explicit, data-driven signal for when a team can take on calculated risk (a risky deployment, a chaos experiment) versus when it should prioritize reliability work — replacing a subjective velocity-vs-reliability argument with an objective, shared metric.

  4. "Why should an external SLA typically be set looser than the internal SLO for the same metric?" Strong answer: To provide a safety margin — if the SLA and SLO were identical, any month the team narrowly missed its own internal target would automatically also breach the external, often contractual/penalized commitment, removing any buffer for normal operational variance.

Chapter summary / cheat sheet

  • SLI: the measured metric. SLO: the internal target. SLA: the external, often contractual commitment (set looser than the SLO for safety margin). Error budget: the allowed unreliability implied by the SLO (100% − SLO%).
  • Each additional "nine" of availability represents roughly a 10x reduction in allowed downtime, typically requiring a qualitatively different, more expensive architecture — never choose a target arbitrarily.
  • The error budget's real value is as an explicit, data-driven tool for deciding when to prioritize velocity (calculated risk-taking) versus reliability work — replacing subjective argument with a shared, objective metric.
  • Precision in this vocabulary matters because each term plays a genuinely distinct role in what's being measured, targeted, or externally committed to.