Chapter 4: Horizontal vs. Vertical Scaling, and Load Balancing/Auto-Scaling

Why this matters

The companion System Design book's Chapter 6 covered horizontal/vertical scaling and load balancing mechanics in depth. This chapter's specific angle: scaling isn't purely a capacity/performance concern — it's directly a reliability concern, since under-provisioned capacity is itself a reliability failure (a system that falls over under real load has failed just as surely as one felled by a server crash), and over-provisioned capacity is a cost-efficiency failure with its own organizational consequences.

Scaling as a reliability property, not just a performance one

[SENIOR INSIGHT] A system that works correctly at low load but degrades or fails entirely under real, expected peak load has a genuine reliability problem, even though nothing about its code is "broken" in the traditional sense — this reframes System Design Chapter 4's capacity estimation as a reliability practice specifically: an SLO (Chapter 2) implicitly includes "remains available and performant under expected peak load," and a capacity shortfall is a direct, predictable path to an SLO breach, deserving the same seriousness as any other reliability risk.

A system with ZERO code bugs that falls over every Friday evening
(predictable peak traffic) has a real reliability problem -- the
ABSENCE of a traditional "bug" doesn't mean the system is reliable;
capacity planning failures are reliability failures too.

Auto-scaling as a reliability mechanism, with real limits

[BEGINNER] Directly building on the companion System Design book's Chapter 6 auto-scaling coverage: auto-scaling is a genuine reliability mechanism (automatically maintaining sufficient capacity as load changes) but has real, reliability-relevant limits worth restating explicitly here — provisioning new capacity takes real time, meaning auto-scaling protects against gradual load growth far more reliably than it protects against a sudden, sharp spike. For predictable spike events (a product launch, a marketing campaign, a known high-traffic date), pre-provisioning ahead of the event is a more reliable strategy than trusting reactive auto-scaling to keep pace.

Setting auto-scaling triggers correctly: leading indicators, not lagging ones

[SENIOR INSIGHT] A common, subtle mistake: triggering auto-scaling based on a metric that lags behind the actual problem, rather than one that leads it. CPU utilization, for instance, might not rise until a system is already struggling under queued, backed-up work — by the time CPU crosses a threshold, request latency may have already degraded for real users. A leading indicator — request queue depth, or the rate of change in incoming request volume — triggers scaling before the lagging symptom (degraded latency, high CPU) actually manifests, giving the new capacity time to come online before users are affected at all.

Lagging trigger: "scale up when CPU exceeds 80%" -- by the time this
                 fires, users may already be experiencing degraded
                 latency, since CPU saturation is often a SYMPTOM of
                 an already-developing capacity problem, not its earliest signal.

Leading trigger: "scale up when request queue depth exceeds N, OR when
                 incoming request rate increases by X% over the last
                 5 minutes" -- catches the capacity problem developing,
                 before it fully manifests as user-visible degradation.

Load balancing's reliability role, restated

[IN PRACTICE] Directly the companion System Design book's Chapter 6 health-check discussion, framed explicitly as a reliability mechanism: a load balancer's health checks are what actually convert "one server instance failed" from a partial outage (that instance's share of users get errors) into a non-event (traffic silently redistributes to healthy instances) — this is precisely why health check configuration (how often, what counts as "unhealthy," how quickly a failed instance is removed from rotation) deserves the same deliberate design attention as any other reliability mechanism in this book, not a default, unexamined configuration.

Common mistakes

1. Treating capacity planning as purely a cost/performance exercise, disconnected from reliability considerations, missing that a capacity shortfall under real peak load is a genuine SLO-breaching reliability failure.
2. Relying purely on reactive auto-scaling for a known, predictable traffic spike, rather than pre-provisioning ahead of the event — repeating the exact System Design book Chapter 6 limitation this chapter restates specifically as a reliability risk.
3. Setting auto-scaling triggers on lagging indicators (CPU, memory) rather than leading ones (queue depth, request rate change), allowing user-visible degradation to occur before new capacity has time to come online.
4. Leaving load balancer health check configuration at an unexamined default, rather than deliberately tuning it (detection speed, failure threshold) to match the system's actual reliability requirements.

Discussion / interview questions

1. "Why should capacity planning be considered a reliability practice, not just a cost/performance one?" Strong answer: An SLO implicitly includes remaining available and performant under expected peak load — a capacity shortfall under real load is a direct, predictable path to an SLO breach, deserving the same seriousness as any other reliability risk, even though nothing in the code is traditionally "broken."

2. "Why might CPU utilization be a poor choice of auto-scaling trigger compared to request queue depth?" Strong answer: CPU utilization is often a lagging indicator — by the time it crosses a scaling threshold, a capacity problem may already be developing and affecting real user latency; a leading indicator like queue depth or request rate change catches the developing problem earlier, giving new capacity more time to come online before users are affected.

3. "Why would you pre-provision capacity for a known, predictable traffic spike rather than relying on auto-scaling alone?" Strong answer: Provisioning new instances takes real time; auto-scaling handles gradual load growth well but can't always react fast enough to a sudden, sharp, known spike — pre-provisioning ahead of a predictable event (a product launch, a scheduled sale) avoids this reaction-time gap entirely.

4. "How does load balancer health check configuration affect a system's actual reliability?" Strong answer: It determines how quickly a failed instance is detected and removed from traffic rotation — a slow or lenient health check configuration extends the window during which users are routed to a failing instance, directly affecting the system's measured reliability (SLI); this configuration deserves deliberate tuning, not a default, unexamined setting.

Chapter summary / cheat sheet

• Capacity planning is a reliability practice, not just a cost/performance exercise — a capacity shortfall under real peak load is a genuine SLO-breaching failure.
• Auto-scaling handles gradual load growth well but has real reaction-time limits against sudden spikes — pre-provision ahead of known, predictable events instead of relying on it alone.
• Set auto-scaling triggers on leading indicators (queue depth, request rate change) rather than lagging ones (CPU, memory), to catch developing capacity problems before they become user-visible.
• Load balancer health check configuration directly determines how quickly a failed instance stops receiving traffic — deserves deliberate tuning, not an unexamined default.

---