Chapter 15: Worked Example: Chat System

Why this matters

The chat system worked example exercises a genuinely different set of skills than the URL shortener — real-time bidirectional communication (directly the companion Networking book's Chapter 6 WebSockets discussion), message ordering and delivery guarantees, and a different, more symmetric read/write pattern.

Step 1: Clarify requirements

Clarifying questions: "One-on-one chat, group chat, or both? Do we need delivery/read receipts? Should messages persist indefinitely, or is this more ephemeral? What's the expected concurrent user scale?"

Assumed answers: both one-on-one and small group chat (up to ~50 members); delivery receipts needed, read receipts optional; messages persist indefinitely; assume 10 million concurrent connected users at peak.

Step 2: Estimate scale

10 million concurrent connections -- each needs a PERSISTENT connection
(directly Chapter 8's async/event-driven reasoning, and the companion
Networking book's Chapter 9 -- this is fundamentally an I/O-bound,
many-mostly-idle-connections problem, ideal for an event-driven server
model rather than one-thread-per-connection).

Assume each active user sends ~20 messages/hour during active use:
10,000,000 x 20 / 3600 ≈ 55,555 messages/second (average, during peak usage)

Message storage: ~200 bytes/message x messages/day
   (10M users x 20 msg/hour x 8 active hours/day ≈ 1.6 billion msg/day)
   x 200 bytes ≈ 320 GB/day -- a genuinely large volume, pointing toward
   needing sharding (Chapter 7) for message storage specifically, unlike
   the URL shortener's much smaller estimated volume.

Step 3: Sketch a high-level design

Client <--WebSocket (persistent connection)--> Connection Gateway Servers
                                                        |
                                        (many gateway servers, each holding
                                         a subset of all active connections)
                                                        |
                                                        v
                                              Message Service
                                                        |
                                        +---------------+---------------+
                                        v                                 v
                              Message Queue (Chapter 8)          Message Database
                              (for delivery to recipient's        (sharded by
                               specific gateway server)            conversation ID)

Step 4: Deep-dive — the hardest part: routing a message to a recipient's specific connection

Why this is the hardest part: "The tricky part here isn't sending or storing a message — it's that with many gateway servers each holding a subset of connections, when User A sends a message to User B, the system needs to know which specific gateway server currently holds User B's connection, since B could be connected to any of potentially thousands of gateway instances."

The solution — a connection registry:

A shared, fast-lookup store (Redis, Chapter 5) maps:
  user_id -> gateway_server_id (which specific server holds their connection)

Message flow:
1. User A sends a message to User B via A's gateway server.
2. Message Service looks up B's CURRENT gateway server in the registry.
3. Message Service publishes the message to a queue/channel specific to
   B's gateway server (Chapter 8's pub/sub pattern -- exactly one gateway
   server, the one actually holding B's connection, needs to receive this).
4. B's gateway server delivers the message over B's existing WebSocket connection.
5. If B is offline (no entry in the registry, or delivery fails): store
   the message as UNDELIVERED, and deliver it once B reconnects (checking
   for undelivered messages on connection establishment).

Handling the registry's own failure mode: "This registry needs to be updated the moment a user connects or disconnects — if a gateway server crashes without cleanly deregistering its connections, the registry could have stale entries. I'd have gateway servers send a periodic heartbeat that refreshes each connection's registry entry with a short TTL (companion Database book's Chapter 7 concept), so a crashed server's stale entries expire automatically within a bounded, short window rather than persisting indefinitely."

Step 5: Discuss trade-offs and failure modes

"I chose sharding the message database by conversation ID specifically because nearly every query (fetch messages for conversation X) is naturally scoped to one conversation — this aligns the shard key with the dominant query pattern (Chapter 7's core lesson), avoiding scatter-gather queries. If a gateway server crashes, its connected users are disconnected and need to reconnect (ideally with automatic client-side reconnection logic) — their messages aren't lost, since the message database is the durable source of truth regardless of gateway server health, only real-time delivery is briefly interrupted for those specific users. I'd also want delivery to be idempotent (Chapter 8) in case of at-least-once message queue redelivery, to avoid a recipient seeing the same message twice."

Common mistakes

1. Not recognizing the need for a connection registry, assuming any gateway server can directly reach any connected user without a lookup mechanism — this is precisely the hardest, most distinguishing part of this specific system's design.
2. Choosing HTTP polling instead of WebSockets for a genuinely real-time chat requirement, missing the companion Networking book's Chapter 6 distinction between request/response and genuinely persistent, bidirectional connections.
3. Sharding the message database by an arbitrary key (like message ID) instead of conversation ID, misaligning the shard key with the dominant query pattern and risking scatter-gather queries for the most common operation (fetching a conversation's message history).
4. Not addressing what happens when a recipient is offline, missing a real, common case (not every message recipient is connected at the moment a message is sent) that any real chat system must handle.

Chapter summary / cheat sheet

• Chat systems require persistent, bidirectional connections (WebSockets) — an inherently different, I/O-bound-at-scale problem from a stateless request/response system like the URL shortener.
• The genuinely hard part: routing a message to a recipient's specific connection when many gateway servers each hold only a subset of all connections — solved via a shared connection registry (user → gateway server) with TTL-based staleness handling.
• Shard the message database by conversation ID, aligning with the dominant "fetch this conversation's messages" query pattern.
• Handle offline recipients (store-and-forward undelivered messages) and idempotent delivery (guarding against at-least-once queue redelivery) as explicit, necessary design considerations.

---