Chapter 6: Load Balancing and Horizontal Scaling

Why this matters

"Just add more servers" is the intuitive answer to a capacity problem, and it's often correct — but horizontal scaling only works if the system is actually designed to support it, and load balancing (the mechanism that makes multiple servers usable as one logical service) has its own real design decisions that determine whether scaling out actually delivers the expected capacity increase.

Horizontal vs. vertical scaling, revisited at the systems level

[BEGINNER] Vertical scaling (a bigger, more powerful single server) is simple — no architectural changes needed — but has a hard ceiling (there's a limit to how large a single machine can get) and creates a single point of failure. Horizontal scaling (more servers, working together) has no hard ceiling and improves fault tolerance (one server failing doesn't take down the whole system), but requires the application to actually support running multiple instances — which is not automatic, and is exactly what this chapter's remaining sections address.

The prerequisite for horizontal scaling: statelessness

[SENIOR INSIGHT] The single most important design requirement for horizontal scaling to work at all: application servers must be stateless — any given request must be servable by any server instance, with no server-specific state (like an in-memory user session) required to handle it. If server-specific state exists, a load balancer routing a user's second request to a different server than their first would lose that state entirely.

Stateful (breaks horizontal scaling):
Server A holds user 42's session in its own memory after login.
Load balancer routes user 42's NEXT request to Server B --
Server B has NO knowledge of user 42's session -- request fails
or the user is unexpectedly logged out.

Stateless (horizontal scaling works correctly):
Session data lives in a shared store (Redis, a database) that
ANY server instance can read -- Server B, receiving user 42's
next request, looks up the session from the shared store and
handles it identically to how Server A would have.

This is precisely why moving to a horizontally-scalable architecture often requires an explicit refactor: extracting session state, in-memory caches, or any other server-local state into a shared, external store that every instance can access identically.

Load balancing algorithms, and matching them to the workload

[BEGINNER] Building on the companion Networking book's Chapter 8 coverage of load balancer mechanics:

• Round robin — simplest, cycles through servers evenly; works well when requests are roughly uniform in cost.
• Least connections — routes to whichever server currently has the fewest active connections; better than round robin when request cost varies significantly (a mix of fast and slow requests), since it accounts for actual current load rather than just request count.
• Consistent hashing — routes based on a hash of some request attribute (often a user ID), so the same user consistently reaches the same server — useful specifically when some server-local state or caching benefit exists (a local, per-server cache that's more effective when the same user's requests consistently land on the same server, even though the session itself should still live in a shared store per the statelessness principle above).

Health checks: removing failed instances automatically

[BEGINNER] A load balancer needs to know which backend instances are actually healthy — a health check is a periodic, automated request (often a simple /health endpoint) the load balancer sends to each instance; an instance failing to respond correctly is automatically removed from the rotation until it recovers, preventing the load balancer from continuing to route real user traffic to a broken instance.

Auto-scaling: horizontal scaling that responds to real-time demand

[IN PRACTICE] Auto-scaling automatically adds or removes server instances based on real-time load metrics (CPU utilization, request queue depth, request rate) — rather than provisioning for peak capacity permanently (wasteful during normal, lower-traffic periods) or provisioning for average capacity permanently (insufficient during peak), auto-scaling lets a system's actual running capacity track its actual real-time demand, directly connecting to Chapter 4's peak-vs-average estimation: a system designed only around average load, with no auto-scaling or peak-provisioning strategy, will predictably fail exactly when it matters most.

Low traffic period: 3 server instances running (sufficient for current load)
Traffic spike begins: auto-scaling detects rising CPU/queue depth,
                       provisions 5 additional instances over the next
                       few minutes
Traffic subsides: auto-scaling detects falling load, gradually
                   de-provisions instances back down toward baseline

A real trade-off worth naming: auto-scaling isn't instantaneous — provisioning a new instance takes real time (booting, initializing, becoming healthy per the health check above), which means a very sudden, sharp traffic spike can outpace auto-scaling's reaction time, causing real, if temporary, degradation before new capacity comes online. This is exactly why systems expecting predictable, sudden spikes (a flash sale, a scheduled event) sometimes pre-provision extra capacity ahead of the known event, rather than relying purely on reactive auto-scaling.

Common mistakes

1. Assuming an application can be horizontally scaled without first verifying it's actually stateless, discovering the session/state problem only after attempting to scale out and encountering broken, inconsistent behavior.
2. Using round robin for a workload with highly variable request costs, missing that least-connections (or a similar load-aware algorithm) would distribute actual load more evenly.
3. Not implementing health checks, allowing a load balancer to keep routing traffic to a failed or degraded instance, worsening user impact instead of automatically routing around it.
4. Relying purely on reactive auto-scaling for a known, predictable traffic spike (a scheduled sale, a major announcement), rather than pre-provisioning capacity ahead of an event whose timing is already known.

Discussion / interview questions

1. "What's the difference between horizontal and vertical scaling, and what's the key prerequisite for horizontal scaling to work?" Strong answer: Vertical scaling makes a single server bigger (simple, but a hard ceiling and a single point of failure); horizontal scaling adds more servers (no hard ceiling, better fault tolerance) but requires the application to be stateless — any server-specific state must be moved to a shared store, or a load balancer routing a user's requests to different servers will lose that state.

2. "How would you choose between round robin and least-connections load balancing?" Strong answer: Round robin fits workloads with roughly uniform request cost; least-connections fits workloads with significant request cost variation, since it accounts for actual current server load rather than just distributing request count evenly.

3. "What is auto-scaling, and what's a real limitation of relying on it purely reactively?" Strong answer: Auto-scaling adjusts running capacity based on real-time load metrics rather than fixed peak or average provisioning; its limitation is that provisioning new instances takes real time, so a very sudden, sharp spike can outpace its reaction time — known, predictable spikes are better handled with pre-provisioned capacity ahead of the event.

4. "Why can't a load balancer simply distribute traffic evenly across servers that maintain their own in-memory session state?" Strong answer: A user's subsequent requests might land on a different server than their first, and that server has no knowledge of the session created on the original server — breaking functionality that depends on that state; session data needs to live in a shared store all instances can access identically for horizontal scaling to work correctly.

Chapter summary / cheat sheet

• Horizontal scaling (more servers) has no hard ceiling and improves fault tolerance, but requires the application to be stateless — the prerequisite most commonly missed when first attempting to scale out.
• Load balancing algorithms: round robin (uniform request cost), least connections (variable request cost, load-aware), consistent hashing (same user consistently to the same server, useful for local caching benefits).
• Health checks automatically remove failed/degraded instances from rotation, preventing continued routing to broken servers.
• Auto-scaling tracks real-time demand rather than fixed peak/average provisioning, but has real reaction-time limits — known, predictable spikes are better handled with pre-provisioned capacity.

---