The previous three posts worked inside one program. From this post on, the unit of design is a whole server, and the questions change: is one machine enough, when do you need two, where does the data live, what happens when a machine dies?
System design is often presented as diagrams: a box for "load balancer", a box for "cache", an arrow to "database". Diagrams are easy to draw, but they can't answer the most important question: do you need those boxes at all? The answer is in the numbers. This post builds the estimating tools first, then goes through each block.
The route: a latency table (mostly measured in the Computers from the inside out series), quick estimates, Little's law and why you shouldn't run a machine near full load, scaling up and out, caches, databases (indexes, replication, sharding), consistency, queues, and handling failure.
Every number in this post was measured on a real machine, with Python 3.12 and SQLite, single-threaded, except those clearly marked as estimates.
A latency table
Every programmer should know the orders of magnitude of these numbers. Most of them were measured in earlier posts on this site:
| Operation | Time |
|---|---|
| Read from the L1 cache | 0.87 ns |
| Read from RAM (random access) | 73 ns |
| Look up a key in a Python dict | ~30 ns |
One system call (write) |
345 ns |
| Read one SQLite row by primary key | 1.4 µs |
Create a process (fork) |
267 µs |
| Scan 200,000 SQLite rows without an index | 2.8 ms |
| One round trip over the network to the nearest Cloudflare server | 55–99 ms |
| Open a new HTTPS connection (TCP + TLS) | ~145 ms |
The first and last rows are 8 orders of magnitude apart, a hundred million times. The consequences for design:
- One round trip over the network costs as much as tens of thousands of indexed database reads. A page that calls 20 other services one after another will be slow because of the network, not the code.
- In-process memory is tens of times faster than a database on the same machine, and a database on the same machine is faster than anything across the network.
- A query without an index can turn a microsecond operation into milliseconds.
Quick estimates
Before choosing an architecture, estimate the load. Say an application has 1 million daily active users, each making 20 requests:
20 million requests / day ÷ 86,400 seconds ≈ 230 requests/second on average
peak hours are usually 3–5 times that ≈ 700–1,200 requests/second
(This is an estimate, not a measurement; the point is to know whether you're at 100, 10,000 or 1,000,000 requests per second.) At a few hundred to a thousand requests per second, one good server running a carefully written web application is usually enough, as long as each request doesn't do anything too heavy. Most systems in the world never go beyond this level. Knowing that helps you avoid building a 20-machine cluster for a load one machine can carry.
Another useful habit: write down the read-to-write ratio. A news site, a blog, a link shortener read hundreds of times more than they write, and their architecture revolves around making reads cheap.
Little's law, and why not to run near full
Little's law: the number of requests in the system equals the rate at which they arrive times how long each one stays:
L = λ × W
230 requests/second × 0.05 seconds = 11.5 requests in progress at once
The law holds for any stable system, whatever its insides, and it tells you how many database connections, how many workers, how much memory you need for requests in flight.
The next question: a server takes 10 ms on average per request, so it can handle 100 requests per second. Should you run it at 95 requests per second? Simulating a simple queue (requests arrive at random, processing time random around 10 ms, one machine handling them in turn), 200,000 requests at each load level:
| Load | Average time (waiting + processing) | Theory | p99 |
|---|---|---|---|
| 50% | 20.0 ms | 20.0 ms | 91 ms |
| 70% | 33.2 ms | 33.3 ms | 153 ms |
| 80% | 50.5 ms | 50.0 ms | 225 ms |
| 90% | 96.3 ms | 100.0 ms | 416 ms |
| 95% | 203.4 ms | 200.0 ms | 939 ms |
The "theory" column is the formula for an M/M/1 queue: time = processing time ÷ (1 − load). The simulation matches almost exactly. The scary part is the shape: from 50% to 90% load, the time grows almost 5 times; from 90% to 95%, it doubles again, and the slowest 1% of requests take almost a second. Near full, even a slight rise in load makes the queue balloon.
That's why real systems run at about 50–70% of capacity at peak, and why "the CPU's only at 80%" is a worrying sentence.
Scaling up and scaling out
When one machine isn't enough, there are two directions:
- Vertical (scale up): a bigger machine, more cores, more RAM. The simplest option, no code changes, but there's a ceiling, and one machine is still a single point of failure.
- Horizontal (scale out): many identical machines, with a load balancer in front sharing out requests. Almost no ceiling, and when one machine dies, the others take over.
Scaling out is only easy when application servers are stateless: everything that must be remembered between requests (login sessions, shopping carts) lives in a shared database or cache, not in one machine's memory. Then any request can go to any machine, and adding machines adds capacity. The part with state, the database, is the hard part to scale, and most of system design is about it.
Caches
A cache keeps a copy of data somewhere closer and cheaper to read. It works because real-world access is uneven: a few keys are asked for a lot, most keys rarely. A common distribution is Zipf: the k-th most popular key is requested with frequency proportional to 1/k.
Simulating 300,000 requests over 100,000 keys following a Zipf distribution, with an LRU cache (evict the least recently used key when full) at four sizes:
| Cache size | Share of keys | Hit rate |
|---|---|---|
| 100 keys | 0.1% | 29.0% |
| 1,000 keys | 1% | 50.7% |
| 10,000 keys | 10% | 72.9% |
| 50,000 keys | 50% | 84.0% |
Keeping 1% of the keys stops half the requests. That's why caches are worth so much: a small cache layer in front of a database can halve its load. But the returns diminish: growing the cache from 10% to 50% of the keys, 5 times the memory, adds only 11 percentage points.
Caches sit at many levels: the browser, a CDN (servers at the edge of the network, close to users), in-application caches, a cache in front of the database (Redis, Memcached), and the database's own buffer. This site uses the first two through the Cache-Control header:
# /static/...?v=<content hash>: the URL changes whenever the file does, so it can be cached forever.
STATIC_VERSIONED = "public, max-age=31536000, immutable"
# HTML page shells (no per-user or live data in them): short browser cache, longer edge cache.
PAGE = "public, max-age=60, s-maxage=300, stale-while-revalidate=600"
The hard part of caching is invalidation: when the original data changes, the copy must change too. Three common approaches: an expiry time (a TTL, like max-age=60: accept data up to 60 seconds old), deleting actively on write, and putting a version into the key (?v=<content hash> above: when the file changes the URL changes, and the old copy never has to be deleted; nobody asks for it any more).
Databases: indexes
A table of 200,000 users in SQLite, searched by email, before and after creating an index:
db.execute("select id from users where email = ?", (email,))
# before: SCAN users
db.execute("create index ix_email on users(email)")
# after: SEARCH users USING COVERING INDEX ix_email (email=?)
| Per query | |
|---|---|
| No index (scan the whole table) | 2.80 ms |
| With an index (B-tree) | 0.6 µs |
About 4,700 times faster, and building the index for 200,000 rows took just 21 ms. A table scan grows linearly with the number of rows; a B-tree index only grows with the log of the number of rows, so at 20 million rows the gap is far larger. EXPLAIN QUERY PLAN (or EXPLAIN in Postgres and MySQL) is the first tool to reach for when a query is slow: the word SCAN on a large table is almost always the problem.
Indexes aren't free: every write has to update the table and all of its indexes, and indexes take space. Index the columns you actually search on, not every column.
Databases: replication and sharding
When one database isn't enough:
- Replication: one primary takes every write, ships the changes to a few replicas, and reads are spread across the replicas. A good fit for read-heavy systems. The price is replication lag: a replica can be milliseconds to seconds behind the primary, so a user who just edited their profile and reads from a replica may see the old version. The usual fix: read what a user has just written from the primary.
- Sharding: split the data by a key (say, user ID) across several databases, each holding a part. It scales writes too, but queries across shards, transactions across shards, and resharding when adding shards all become hard. Sharding is the step to take last, when everything else has run out.
Consistency
When data has several copies on several machines, and the network between them can break, the system has to choose. The CAP theorem says: when the network is partitioned (P), a system can keep only one of two things, consistency (C, everyone reads the same latest value) or availability (A, every request gets an answer). A bank chooses consistency: better to refuse a transaction than let money be withdrawn twice. A social network chooses availability: a like count that shows up a few seconds late is fine.
Even when the network is fine, there's a trade-off between latency and consistency: waiting for every copy to confirm before answering means waiting for the slowest machine. Most systems mix both: strong consistency for money and stock levels, "eventual consistency" for counters, feeds and statistics.
Queues
Not everything has to be finished before answering the user. Sending email, making thumbnails, recording statistics can go into a message queue for workers to do later. The benefits:
- Faster requests: the user doesn't wait for the SMTP server. This site does exactly that on a small scale, with
BackgroundTasks(seen in the previous post): "send in the background, so a slow SMTP server never holds up the page". - Absorbing bursts: 10,000 jobs arriving at once wait in the queue instead of knocking the workers over.
- Decoupling: the sender doesn't need to know who the receivers are or how many there are (Observer at the scale of a system, called pub/sub).
The price: results arrive later, and you have to allow for a message being processed twice (a worker dies halfway, the message is redelivered). So processing should be idempotent: doing it twice gives the same result as doing it once.
Handling failure
At the scale of many machines, failure isn't an exception; it's a certainty. A few minimum rules:
- Always have a timeout. A call without one can hang forever and take a worker with it. This site's code sets a timeout on every outside call: 10 seconds for the OAuth providers, 20 seconds for SMTP.
- Retry with a limit, with backoff, with randomness. Retrying immediately and without limit sends an overloaded service more requests exactly when it's weakest. Use growing delays (exponential backoff) plus a little randomness (jitter) so thousands of clients don't all retry at once.
- Idempotency keys for operations that mustn't happen twice (charging money, creating an order): the client sends a unique key, the server remembers keys it has processed and returns the earlier result if it sees one again.
- Graceful degradation: when the product recommendations die, the product page still shows, just without recommendations.
Summary
| Block | Number / key idea |
|---|---|
| Latency table | 8 orders of magnitude from L1 (0.87 ns) to a new HTTPS connection (~145 ms) |
| Estimates | 1 million users × 20 requests ≈ 230 requests/second, ~1,000 at peak |
| Little's law | Requests in progress = arrival rate × time in the system |
| Load | 50% → 20 ms, 90% → 96 ms, 95% → 203 ms (p99 939 ms) |
| Scaling | Stateless application servers; the database is the hard part |
| Caches | 1% of keys stops ~51% of requests; invalidation is the hard part |
| Indexes | 2.80 ms → 0.6 µs (~4,700 times) on 200,000 rows |
| Replication / sharding | Read-heavy means replicate; sharding comes last |
| Consistency | CAP: when the network breaks, choose consistency or availability |
| Queues | Fast requests, absorb bursts; processing must be idempotent |
| Failure | Timeouts, retries with backoff + jitter, idempotency keys |
That's enough blocks. The last post of the series puts them together into a system of many services, with a present-day condition: most of the code is written by AI agents. The example is a concert ticketing system: seat selection, holds, payment, tickets, notifications, check-in and an on-sale waiting room. What people have to provide, how to split the services, measuring which way of writing the code sells seats twice, and how to write documentation the AI can read, understand and extend features from.