Chapter 7: Resilience Patterns: Timeouts, Retries, Circuit Breakers, Bulkheads, Load Shedding

Why this matters

Timeouts, retries with backoff/jitter, and circuit breakers were covered in depth in the companion Networking book's Chapter 11. This chapter briefly recaps those (since they're foundational resilience patterns this chapter's new material builds alongside) and then introduces bulkheads and load shedding — two patterns not yet covered in this series, both essential for a complete resilience toolkit at real production scale.

Quick recap: timeouts, retries, circuit breakers

[BEGINNER] Timeouts bound how long to wait for a response before giving up (companion Networking book, Chapter 11). Retries with exponential backoff and jitter retry a failed operation with progressively longer, randomized delays, avoiding a retry storm. Circuit breakers stop sending requests to an already-failing dependency for a cooldown period, preventing cascading failure. All three remain foundational — this chapter's contribution is two patterns that address failure modes these three don't fully cover.

Bulkheads: isolating failure to one part of a system

[BEGINNER] Named after a ship's bulkheads — physical compartments that prevent water from flooding the entire vessel if one section is breached — a software bulkhead isolates resources (thread pools, connection pools, memory) allocated to different parts of a system, so that one part's failure or resource exhaustion can't consume resources needed by unrelated parts.

Without bulkheads: ONE shared thread pool serves requests to Dependency A
                    (fast, healthy) AND Dependency B (currently slow/failing).
                    Requests to B pile up, consuming more and more threads
                    from the SHARED pool while waiting -- eventually,
                    there are no threads left for requests to A either,
                    even though A itself is completely healthy. B's
                    failure has taken down A's functionality too.

With bulkheads: SEPARATE, dedicated thread pools for A and B. B's
                slowness/failure can exhaust B's OWN pool, but A's pool
                remains completely unaffected -- A's functionality
                continues working normally despite B's ongoing failure.

[SENIOR INSIGHT] This is directly the same principle as a ship's physical bulkheads, and it's a distinct, complementary pattern from a circuit breaker: a circuit breaker stops sending requests to a specific failing dependency; a bulkhead prevents that dependency's failure from consuming shared resources needed by unrelated parts of the system in the first place, even before a circuit breaker might trip. Using both together is common and often necessary — a circuit breaker alone doesn't help if resource exhaustion from the failing dependency has already happened by the time enough failures accumulate to trip it.

Load shedding: deliberately rejecting some requests to protect the system as a whole

[BEGINNER] Load shedding is deliberately rejecting some incoming requests (typically with an immediate, cheap error response) when a system is under more load than it can healthily handle — rather than attempting to serve every request and having all of them degrade or fail as the system becomes progressively more overwhelmed.

Without load shedding: as load exceeds capacity, EVERY request's
                        latency degrades progressively -- eventually,
                        the entire system becomes unresponsive for
                        EVERYONE, including requests that arrived
                        early and could have been served successfully
                        if the system hadn't been trying to also serve
                        every later, excess request simultaneously.

With load shedding: once load exceeds a defined capacity threshold,
                     EXCESS requests are immediately rejected (a cheap,
                     fast "503 Service Unavailable, try again shortly"
                     response) -- the requests that ARE accepted continue
                     to be served at normal, healthy latency, rather
                     than everyone's experience degrading together.

[SENIOR INSIGHT] Load shedding is a deliberate, explicit trade-off: some users get an immediate, clear failure instead of a slow, degraded, uncertain experience — and this is usually the better outcome for overall system health, since a system serving 80% of its requests quickly and correctly while cleanly rejecting the excess 20% is more genuinely reliable (and often recovers faster once the load spike passes) than a system attempting to serve 100% of requests so slowly and unreliably that effectively none of them succeed within any reasonable timeout. Load shedding is frequently combined with prioritization — shedding lower-priority requests (a non-critical analytics call) first, while continuing to serve higher-priority ones (an actual checkout/payment request) as long as possible.

Common mistakes

1. Using a single, shared resource pool (threads, connections) across dependencies with very different reliability profiles, allowing one failing/slow dependency to exhaust shared resources and take down unrelated, otherwise-healthy functionality.
2. Relying on a circuit breaker alone without bulkheads, missing that resource exhaustion from a failing dependency can occur before enough failures accumulate to actually trip the circuit breaker.
3. Not implementing load shedding, attempting to serve every request regardless of actual system capacity, resulting in universal, catastrophic degradation under overload rather than a controlled, partial rejection that preserves service for the requests that can actually be served well.
4. Shedding load without any prioritization, treating a critical checkout request identically to a non-critical background analytics call when deciding what to reject first under overload.

Discussion / interview questions

1. "What is a bulkhead pattern, and how does it differ from a circuit breaker?" Strong answer: A bulkhead isolates resources (thread pools, connections) allocated to different parts of a system, so one dependency's failure/slowness can't exhaust resources needed by unrelated parts; a circuit breaker stops sending requests to a specific failing dependency once enough failures accumulate — they're complementary, since a bulkhead can prevent resource exhaustion that would otherwise occur before a circuit breaker trips.

2. "What is load shedding, and why might deliberately rejecting some requests actually improve overall system reliability?" Strong answer: Load shedding immediately rejects excess requests beyond a defined capacity threshold, rather than attempting to serve everyone and having every request degrade together — a system serving a subset of requests quickly and correctly while cleanly rejecting the rest is more genuinely reliable than one where universal overload causes effectively all requests to fail via timeout anyway.

3. "How would you decide which requests to shed first under overload?" Strong answer: Prioritize by business criticality — shed lower-priority requests (non-critical background operations, analytics calls) first, continuing to serve higher-priority ones (checkout, payment processing) as long as possible, rather than treating all requests as equally sheddable.

4. "Why might a system with a well-configured circuit breaker still experience a cascading failure from one dependency?" Strong answer: If the failing dependency shares a resource pool (threads, connections) with unrelated parts of the system, that shared resource can be exhausted before enough failures accumulate to trip the circuit breaker — a bulkhead, isolating that dependency's resource usage, is needed to prevent this specific failure mode.

Chapter summary / cheat sheet

• Timeouts, retries with backoff/jitter, circuit breakers (companion Networking book, Chapter 11) remain foundational resilience patterns.
• Bulkheads isolate resource pools per dependency/component, preventing one failing part from exhausting shared resources needed by unrelated, healthy parts — complementary to, and often necessary alongside, circuit breakers.
• Load shedding deliberately rejects excess requests under overload, protecting the requests that CAN be served well rather than letting universal degradation take down effectively everyone — often combined with priority-based shedding.
• A complete resilience toolkit uses several of these patterns together, since each addresses a genuinely distinct failure mode that the others don't fully cover.

---