Systems & Architecture

TL;DR

Every system lives somewhere on a ladder that starts at one thread in one process and ends at a product spread across regions. Each rung adds a new primitive — thread, socket, queue, consensus — and a new failure mode that the rung below didn't have. The mental model at each rung is different; the ladder is the same. Learn the rungs; you'll recognize every new problem as a return to a rung you've seen.

You will be able to

• Name the six rungs and the primitive each one introduces.
• For any system, identify which rung its current problem lives at.
• Predict the failure mode a proposed design inherits from its highest rung.

The Map

• You will be able to
• The Map
• Rung 1 — One process, one thread
• Rung 2 — One process, many threads
• Rung 3 — One host, many processes
• Rung 4 — Many hosts, one network
• Rung 5 — Many services, one system
• Rung 6 — Many regions, one product
• How the rungs connect
• Standards & Specs
• Test yourself

Every rung is a superset of the previous — you inherit all of the lower failure modes plus the new one. A region-replicated system can still have a deadlock on a single host.

Rung 1 — One process, one thread

The baseline: a program with a call stack, a heap, system calls. No concurrency. Failure modes are the ones we teach first: logic bugs, infinite loops, OOM, file-not-found.

     ┌─────────────────────────────────────┐
     │            PROCESS                  │
     │  ┌────────────┐   ┌──────────────┐  │
     │  │   stack    │   │     heap     │  │
     │  │ ────────── │   │              │  │
     │  │   main()   │   │   new Foo()  │  │
     │  │   f()      │   │   new Bar()  │  │
     │  │   g()      │   │              │  │
     │  └────────────┘   └──────────────┘  │
     │         │                           │
     └─────────┼───────────────────────────┘
               ▼ syscalls: read, write, mmap, …
             KERNEL

The model you want: the stack is sequential, the heap is shared with future self, the kernel is the only other party. If you understand what's on the stack vs heap at any moment, you debug single-threaded bugs quickly.

Go deeper: Bryant & O'Hallaron Computer Systems: A Programmer's Perspective chapters 7–8; strace your own program for 30 minutes and watch the syscalls.

Rung 2 — One process, many threads

Add concurrent execution inside one process. Threads share the heap but not the stack. Suddenly the program has to reason about visibility (did my write reach the other thread?), atomicity (did this read-modify-write happen as one step?), and ordering (which event happens before which?).

     ┌─────────────────────────────────────────────────────┐
     │                    PROCESS                          │
     │                                                     │
     │  ┌─────────┐  ┌─────────┐  ┌─────────┐              │
     │  │ stack A │  │ stack B │  │ stack C │              │
     │  └────┬────┘  └────┬────┘  └────┬────┘              │
     │       │            │            │                   │
     │       └────────────┼────────────┘                   │
     │                    ▼                                │
     │             ┌─────────────┐                         │
     │             │ shared heap │   ← races live here     │
     │             └─────────────┘                         │
     └─────────────────────────────────────────────────────┘

The model you want: every shared mutable word is a potential race. The fix menu is short: make it immutable, make it thread-local, or put a lock around every access path. The language's memory model (Java's happens-before, C++'s sequential consistency, Go's race detector) is the contract between you and the hardware.

Go deeper: Goetz Java Concurrency in Practice chapters 2–5; the CPython GIL deep-dive by David Beazley; run ThreadSanitizer over your own C++ code.

Rung 3 — One host, many processes

Spawn a second process. Now you need IPC — pipes, shared memory, Unix sockets, signals. Processes don't share memory by default, which is both a safety win (no data races across processes) and an efficiency loss (every message is a copy unless you use shared memory).

New failure modes:

• Zombies — child exited, parent didn't wait. Fills the process table.
• Orphans — parent died, child reparented to init. Runs forever unless it watches getppid().
• IPC stalls — pipe buffer full; writer blocks forever because reader died quietly.

The model you want: IPC is slower than memory access but safer than shared memory. When you need speed, shared-memory plus a lock. When you need safety, messages plus careful draining.

Go deeper: Stevens Advanced Programming in the UNIX Environment chapters 15–17; build a minimal one-host job queue using only fork, pipe, and poll.

Rung 4 — Many hosts, one network

Add the network. Now you have partial failure — a message you sent may have arrived, may not have, may have been processed but the ack was lost. You cannot tell which. Every guarantee past this rung is about closing that ambiguity.

New primitives: socket, timeout, retry, heartbeat. New failure modes: partitions (subset of hosts unreachable from another subset), black holes (traffic accepted but silently dropped), flapping (intermittent connectivity that looks like health).

The model you want: every network call has four outcomes: success, failure, timeout, and partial success you'll never be told about. Design for all four, not three.

Go deeper: TCP/IP Illustrated chapters 17–18; Peter Bailis's "Partitions for everyone!" talk; Aphyr's Jepsen series, even the old ones.

Rung 5 — Many services, one system

Services call services. Queues appear. The system becomes a graph, and failures propagate along edges.

New primitives: message queue, consumer group, idempotency key, circuit breaker, backpressure. New failure modes:

• Cascading failure — Svc3 slows down, Svc1's threads pile up waiting, Svc1 becomes unhealthy, Gateway's retries make it worse.
• Retry amplification — upstream retry policy × downstream retry policy × consumer retry policy = 1000x load on the thing that's already sick.
• Ordering bugs — events delivered out of order because queue partitions changed, because a consumer was rebalanced.

The model you want: every cross-service call is a possible timeout, every timeout is a possible retry, every retry must be idempotent. The Foundations handbook covers these primitives formally; this rung is where you feel them.

Go deeper: Nygard Release It! second edition in full; the Netflix Hystrix wiki (Hystrix is dead, the vocabulary survived); the Amazon Builders' Library article on timeouts, retries, and backoff.

Rung 6 — Many regions, one product

Put hosts in multiple regions. Now the physics changes: cross-region RTT is 50–200 ms, and WAN links fail more than LAN links. State must be replicated, and replication makes you confront consistency head-on.

  us-east-1              eu-west-1                 ap-south-1
  ┌────────────┐         ┌────────────┐            ┌────────────┐
  │  Primary   │◀───────▶│  Secondary │◀──────────▶│  Secondary │
  │   DB       │  async  │    DB      │   async    │    DB      │
  └─────┬──────┘         └─────┬──────┘            └─────┬──────┘
        │                      │                         │
  ┌─────┴──────┐         ┌─────┴──────┐            ┌─────┴──────┐
  │  Services  │         │  Services  │            │  Services  │
  └────────────┘         └────────────┘            └────────────┘
       ▲                       ▲                         ▲
       │                       │                         │
       └─── DNS / Anycast ◀────┴──── geo-routed ─────────┘

New primitives: consensus (Raft, Paxos), leader election, multi-region write strategies (active-active, active-passive, leader-per-key), snapshot isolation, conflict-free replicated data types (CRDTs). New failure modes:

• Split brain — a partition makes two regions each believe they're the leader. Both accept writes. Reconciliation is hell.
• Stale reads — you read a region that hasn't caught up yet. "I just saved this — why don't I see it?"
• Asymmetric traffic — one region's users generate writes that the others have to apply asynchronously; failover moves write traffic to a region not provisioned for it.

The model you want: every replication strategy trades between latency, consistency, and availability along exactly the axes CAP/PACELC names. You don't escape the tradeoff; you pick which side you're optimizing.

Go deeper: Kleppmann DDIA chapters 5 and 9; the Spanner paper; Patterns of Distributed Systems by Unmesh Joshi; any Jepsen report on a database you actually run.

How the rungs connect

Climbing is superset-style: each rung inherits everything below plus a new failure mode. You cannot skip rungs in production — you pay for the primitives of the rung you're on, whether or not you understand them.

A region-replicated service with a race condition on one host is still a buggy service. The rung framing is not an excuse to ignore the lower rungs; it's a reminder that failure modes accumulate.

Standards & Specs

• POSIX — IEEE Std 1003.1 — the OS-level primitives at Rungs 1–3 (processes, signals, sockets, file descriptors).
• RFC 9293 — TCP (and its predecessor RFC 793) — the transport every Rung-4 system is secretly negotiating with.
• RFC 9110 — HTTP semantics — the application-layer contract most Rung-4 calls ride on.
• RFC 9000 — QUIC and RFC 9114 — HTTP/3 — the modern transport; relevant whenever you care about head-of-line blocking.
• RFC 8446 — TLS 1.3 — because no service mesh is real without it.
• Raft, Ongaro & Ousterhout (2014) — the consensus algorithm that actually gets implemented (Rung 6).
• Paxos Made Simple, Lamport (2001) — the conceptual ancestor; hard to implement, easy to cite.
• Spanner paper (OSDI 2012) — the reference for global-scale externally consistent databases.
• Lamport, "Time, Clocks, and the Ordering of Events" (1978) — why happens-before is the only order that matters across a partition.
• AWS Builders' Library — Timeouts, retries, and backoff — the production wisdom that should be taped to every Rung-5 design doc.
• Books — Nygard, Release It! (2nd ed); Kleppmann, DDIA; Joshi, Patterns of Distributed Systems; Bryant & O'Hallaron, CS:APP.

Test yourself