In First-principles thinking there's a sentence that sounds simple: "one FastAPI process can hold tens of thousands of waiting requests without tens of thousands of threads". This post takes that sentence apart from the top down: how processes, threads and coroutines differ, what the GIL is, what await really does, who wakes up a sleeping coroutine, and then back to FastAPI to see how it all fits together.
Every number here was measured on a real machine: Linux (WSL2), 12 cores, Python 3.12 and the free-threaded Python 3.14 build.
Three ways to do many things at once
First, two words that often get mixed up:
- Concurrency: many tasks in progress at the same time. One cook with three pots: put one on the stove, and while it heats up, chop vegetables for another.
- Parallelism: many tasks actually running at the same moment. Three cooks, three pots.
A web server needs both: concurrency to hold many waiting requests, parallelism to use many CPU cores. Python has three tools:
| Process | Thread | Coroutine | |
|---|---|---|---|
| Scheduled by | The operating system | The operating system | The event loop (Python code) |
| Memory | Fully separate | Shared within the process | Shared within the thread |
| Runs in parallel on many cores | Yes | Not with default Python (GIL) | No |
| Cost of each | A whole Python interpreter | Its own stack, usually 8 MB of virtual memory on Linux | A few KB |
| Switching | Done by the kernel, the most expensive | Done by the kernel | A function call in Python |
The key point: the operating system only knows about processes and threads. Coroutines are Python's own idea; the kernel has no idea they exist.
CPython and the GIL
CPython, the most common Python, compiles code to bytecode and runs it instruction by instruction in a loop written in C. Every Python object has a reference count that says when it can be freed. If two threads changed that count at the same moment, it could go wrong and corrupt memory. Instead of locking every object, CPython uses one lock for the whole interpreter: the GIL (Global Interpreter Lock). Only the thread holding the GIL may run bytecode.
The GIL is released in three cases:
- When a thread makes an I/O call that may wait, like reading a socket or a file, or
time.sleep. - When a library written in C releases it while computing, like numpy, Pillow or hashlib.
- When a thread has held it for 5 ms while another thread is waiting:
sys.getswitchinterval()returns0.005.
So threads help when waiting on I/O, but not when using the CPU in Python code. Here's a pure CPU task, run 4 times:
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
def count_down(n: int) -> None:
while n:
n -= 1
N = 20_000_000
if __name__ == "__main__":
start = time.perf_counter()
for _ in range(4):
count_down(N)
print(f"one after another : {time.perf_counter() - start:.2f} s")
for name, pool in (("4 threads", ThreadPoolExecutor), ("4 processes", ProcessPoolExecutor)):
start = time.perf_counter()
with pool(max_workers=4) as executor:
list(executor.map(count_down, [N] * 4))
print(f"{name:<18}: {time.perf_counter() - start:.2f} s")
| How it runs | Python 3.12 (GIL) | Python 3.14t (no GIL) |
|---|---|---|
| One after another | 1.39 s | 1.04–1.16 s |
| 4 threads | 1.39–1.46 s | 0.32–0.36 s |
| 4 processes | 0.43–0.49 s | 0.38–0.49 s |
With the GIL, 4 threads are no faster than running one after another; only 4 processes use 4 cores. The free-threaded build (the t suffix, officially supported since Python 3.14 but opt-in) lets threads truly run in parallel. Processes still pay to start a new interpreter for each worker, so they're not 4× faster.
The takeaway for default Python: threads for waiting on I/O, processes for CPU work. Coroutines are a third way to wait on I/O, much cheaper than threads.
Coroutines: functions that can pause
Calling an async def function runs nothing; it only creates a coroutine object. That object is like a function that can stop halfway, remember where it stopped, and continue when told to. Let's drive one by hand, without asyncio:
import types
@types.coroutine
def wait_for(what):
"""Pause the coroutine and hand `what` to whoever is driving it."""
reply = yield what
return reply
async def handler():
print("handler: asking for the user")
user = await wait_for("SELECT name FROM users WHERE id = 1")
print("handler: got", user)
return f"<h1>Hello {user}</h1>"
coro = handler() # nothing runs yet: calling an async function only creates the coroutine
request = coro.send(None) # run until the first pause
print("driver: coroutine is paused, waiting for:", request)
try:
coro.send("An") # resume it with the answer
except StopIteration as done:
print("driver: finished with", done.value)
handler: asking for the user
driver: coroutine is paused, waiting for: SELECT name FROM users WHERE id = 1
handler: got An
driver: finished with <h1>Hello An</h1>
await doesn't wait for anything by itself. It passes a request outward, through every layer of await, to whoever is driving the coroutine with send(). The driver decides when to resume it and with what answer. In asyncio, the driver is the event loop, and what gets passed out is a Future: a promise to report back when the result is ready.
The event loop: write one in 40 lines
An event loop does two things over and over: run the coroutines that are ready until they pause, then ask the operating system "which of these sockets has data now?". Here's a toy version, complete enough to actually run:
import selectors
import socket
import threading
import time
import types
from collections import deque
selector = selectors.DefaultSelector() # EpollSelector on Linux
ready = deque() # coroutines that can run right now
@types.coroutine
def readable(sock):
yield sock # pause, telling the loop which socket we're waiting on
def run(*coroutines):
ready.extend(coroutines)
while ready or selector.get_map():
while ready:
coro = ready.popleft()
try:
sock = coro.send(None) # run it until its next pause
selector.register(sock, selectors.EVENT_READ, coro)
except StopIteration:
pass # this coroutine is done
if selector.get_map():
for key, _ in selector.select(): # epoll_wait: the thread sleeps in the kernel here
selector.unregister(key.fileobj)
ready.append(key.data) # its socket has data: that coroutine can go on
async def fetch(name, delay):
ours, server = socket.socketpair() # stands in for a connection to a slow server
threading.Timer(delay, server.send, [f"reply after {delay} s".encode()]).start()
await readable(ours)
print(f"{name}: {ours.recv(100).decode()}")
start = time.perf_counter()
run(fetch("A", 1.0), fetch("B", 1.0), fetch("C", 0.5))
print(f"total: {time.perf_counter() - start:.2f} s")
C: reply after 0.5 s
A: reply after 1.0 s
B: reply after 1.0 s
total: 1.00 s
Three waits of 1 s, 1 s and 0.5 s take 1 s in total, on a single thread. The most important line is selector.select(): on Linux, that's the epoll_wait system call. Real asyncio does the same things, plus timers, cancellation, error handling and many optimizations.
Down to the kernel: who wakes a coroutine up?
Let's trace the system calls of a real coroutine connecting to a server that deliberately answers after 1 second:
import asyncio
async def fetch():
reader, writer = await asyncio.open_connection("127.0.0.1", 9099)
print("waiting...", flush=True)
data = await reader.read(100) # the coroutine suspends here
print("woke up with", data, flush=True)
asyncio.run(fetch())
Run under strace (trimmed; the first column is the time):
07.756 epoll_create1(EPOLL_CLOEXEC) = 3 the loop creates an epoll instance
07.760 connect(6, ...:9099) = -1 EINPROGRESS non-blocking: returns at once
07.760 epoll_ctl(3, EPOLL_CTL_ADD, 6, EPOLLOUT) "tell me when connected"
07.762 epoll_ctl(3, EPOLL_CTL_ADD, 6, EPOLLIN) await reader.read(): "tell me when there's data"
07.763 write(1, "waiting...")
07.763 epoll_wait(3, [...], 2, -1) the thread sleeps in the kernel...
08.762 recvfrom(6, "hello") = 5 ...and runs again exactly 1 s later
08.762 write(1, "woke up with b'hello'")
While it waits, /proc/<pid>/stat shows the process in state S (sleeping), and the whole program used just 3 CPU ticks. The thread isn't asking "is there data yet?" over and over; it lies still in the kernel. Step by step:
- The coroutine pauses (Python). The socket is non-blocking (
O_NONBLOCK), so when there's no data the kernel returnsEAGAINat once instead of making the thread wait. asyncio registers "tell me when fd 6 is readable" withepoll_ctl, creates aFuture, and the coroutine passes that Future outward and stops. It's now just an object in memory. - The event loop goes to sleep (system call). With nothing ready, the loop calls
epoll_wait, with a timeout set to the nearest timer, or-1to wait forever. The kernel puts the thread to sleep on epoll's wait queue; the scheduler stops running it. - Data arrives (kernel). The network card writes the packet into RAM by DMA and raises a hardware interrupt. The kernel processes it through IP and TCP, copies the data into the socket's buffer, and the socket signals "data available". A callback that epoll attached to the socket puts fd 6 on the ready list and wakes the sleeping thread.
- The coroutine continues (back in Python).
epoll_waitreturns[fd 6: EPOLLIN]. The loop callsrecvfrom, which now has data right away, sets the Future's result, and queues the coroutine to run. On the next pass, the loop callssend(), and the coroutine continues from exactly itsawaitline.
So: hardware → interrupt → kernel → epoll_wait returns → event loop → coroutine. The kernel wakes the thread; the event loop wakes the coroutine.
That's also why one thread can hold tens of thousands of connections: it sleeps in a single place, and one epoll_wait returns every socket that's ready. With threads, every connection needs its own sleeping thread, each with its own stack.
Three details worth knowing:
- Waking up from another thread. The run above also has fds 4 and 5: a pair of sockets asyncio creates for itself. When another thread needs to notify the loop, through
call_soon_threadsafe, it writes 1 byte into that pair soepoll_waitreturns. - Disks can't use epoll. epoll rejects regular files, because the kernel considers them always ready. So "async" file reading in Python, like the
aiofileslibrary, actually runs in a thread pool. Theaiosqlitelibrary likewise runs SQLite in its own thread and reports back to the loop with that samecall_soon_threadsafe. Linux hasio_uringfor truly asynchronous disk I/O, but asyncio doesn't use it. - uvloop. A drop-in event loop written in Cython on top of libuv (the I/O library behind Node.js). The mechanism is still non-blocking sockets and
epoll_wait; only the user-space part is faster.
Back up to FastAPI
A FastAPI app run by uvicorn has three layers:
- uvicorn: the ASGI server. It opens the socket, runs the event loop (using uvloop automatically when installed), and parses HTTP with httptools, a parser written in C.
- Starlette: the ASGI framework underneath FastAPI: routing, requests, responses.
- FastAPI: adds data validation with Pydantic, dependency injection and OpenAPI docs.
ASGI is the contract between server and app: the app is an async function that receives scope (the request's details), receive and send. For each request, uvicorn creates a Task on the event loop to run the app. From there, how you write an endpoint decides where it runs:
async defruns directly on the event loop. Everyawaiton I/O hands the loop to someone else.- Plain
defis sent by FastAPI to a thread pool withrun_in_threadpool. AnyIO manages that pool, with a default maximum of 40 threads. Dependencies written asdefgo there too.
The experiment: three endpoints, each waiting 0.1 seconds in a different way.
import asyncio
import time
from fastapi import FastAPI
app = FastAPI()
@app.get("/async")
async def non_blocking():
await asyncio.sleep(0.1) # waits without holding the thread: the loop serves others
return {"ok": True}
@app.get("/sync")
def in_thread_pool():
time.sleep(0.1) # blocks, but FastAPI runs a plain def in its thread pool
return {"ok": True}
@app.get("/mistake")
async def blocks_the_loop():
time.sleep(0.1) # blocks inside async def: the whole event loop stops
return {"ok": True}
Fire 100 requests at once at each endpoint, with a minimal asyncio client:
import asyncio
import time
N = 100
async def get(path: str) -> None:
reader, writer = await asyncio.open_connection("127.0.0.1", 8123)
writer.write(f"GET {path} HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n".encode())
await writer.drain()
await reader.read() # until the server closes the connection
writer.close()
async def main() -> None:
for path in ("/async", "/sync", "/mistake"):
start = time.perf_counter()
await asyncio.gather(*(get(path) for _ in range(N)))
print(f"{N} requests at once to {path:<9}: {time.perf_counter() - start:5.2f} s")
asyncio.run(main())
| Endpoint | 100 requests at once | Why |
|---|---|---|
/async |
0.11–0.12 s | 100 coroutines waiting together on one loop |
/sync |
0.33–0.35 s | 40 threads: 3 rounds × 0.1 s |
/mistake |
10.08–10.09 s | The loop is blocked: 100 × 0.1 s back to back |
/mistake is the most common bug: time.sleep inside async def doesn't hand over the loop, so for that 0.1 second the whole process stands still, including requests to other endpoints. requests.get(), a synchronous database driver or a heavy computation inside async def all cause exactly this. On my first try, the client even timed out before /mistake finished answering.
A few rules that follow:
- Use
async defonly when every I/O inside it can be awaited: an async database driver,httpx.AsyncClient,asyncio.sleep. - For synchronous libraries, write
def, so FastAPI moves the work to the thread pool. With many long waits, 40 threads becomes the limit; you can raise it withanyio.to_thread.current_default_thread_limiter().total_tokens, but switching to an async library is usually better. - Keep CPU-heavy work off the loop: send it to a
ProcessPoolExecutor, or to a separate worker taking jobs from a queue. - One process uses about one core. To use 4 cores, run
uvicorn --workers 4, or one process per pod and add pods. Processes don't share memory, so all state, like login sessions, has to live in the database or Redis.
The website you're reading runs exactly like this: uvicorn with uvloop, async def endpoints, SQLite through aiosqlite, and one worker, since one core is enough for the current traffic.
In short
| Layer | Who | Main job |
|---|---|---|
| Hardware | Network card, interrupts | Signal that data has arrived |
| Kernel | epoll, the scheduler | Put the thread to sleep, wake it when a socket is ready |
| Event loop | asyncio / uvloop | Call epoll_wait, run whichever coroutines are ready |
| Coroutine | Your async def code |
Run until await, hand over the loop, continue when data arrives |
| Thread pool | AnyIO, 40 threads | Run synchronous code (def) without blocking the loop |
| Process | uvicorn workers, pods | Use many CPU cores |
Once you know what each layer does, almost every question about a Python server's performance can be answered by asking: is this work waiting or computing, and what is it holding while it does?
To go deeper still, inside the interpreter itself: Inside CPython: everything is an object, and what import really does.