The previous post ended with a string of questions: who keeps the page tables, who handles page faults, who decides which program runs on which core? The answer is the operating system, or more precisely its core, the kernel. This post goes through what the kernel does, then moves on to a device that today matters as much as the CPU: the GPU.
The route: the kernel and user space, the cost of a system call, processes, threads and scheduling, devices and drivers, then CPUs and GPUs: why they're designed so differently, measured with the same matrix multiplication, and when the GPU isn't faster.
Every number in this post was measured on a real machine, in C (clang), Python 3.12 and PyTorch 2.14. The GPU here is the integrated GPU in a laptop's chip, used through PyTorch's MPS backend; discrete GPUs in AI servers are far more powerful, but the phenomena are the same. Each measurement runs for a few milliseconds to a few seconds.
The kernel and user space
A CPU has (at least) two modes. In kernel mode, code may do anything: change page tables, talk to hardware, stop any program. In user mode (user space), where every ordinary program runs, those instructions are forbidden: the CPU refuses to run them and tells the kernel. A program can't open a file, send a network packet or get more memory by itself; it has to ask the kernel, through a system call.
A system call is a special machine instruction (svc on arm64, syscall on x86-64): the program puts the number of the job it wants and its arguments into registers, runs that instruction, and the CPU jumps into the kernel at a fixed address. The kernel checks the request, does the work, then returns to the program. The cost of that trip:
for (int i = 0; i < n; i++) x = plain_function(x); // an ordinary function call
for (int i = 0; i < n; i++) write(fd, &c, 1); // a real trip into the kernel and back (fd = /dev/null)
| Each time | |
|---|---|
| An ordinary function call | 0.4 ns |
A write() system call of 1 byte to /dev/null |
345 ns |
os.write() from Python |
372 ns |
os.getpid() from Python |
11 ns |
Writing one byte to /dev/null, a "file" that throws away everything written to it, meaning there's almost nothing to do, still costs almost 900 times a function call. Most of the cost is the trip itself: switching CPU mode, saving and restoring registers, checking arguments, and the defences against Spectre-style vulnerabilities from part 2 (scrubbing CPU state when crossing the kernel boundary). os.getpid() is unusually cheap: 11 ns is too fast for a trip into the kernel, which suggests the C library answers from a saved copy.
The most familiar practical consequence: batch before you call. Python files are buffered by default: f.write("a") a thousand times doesn't cause a thousand system calls, but gets gathered into a few big writes. It's the same idea met many times in the earlier series: grouping rows into chunks when streaming (the Python series), gathering many users' tokens into one batch (the LLM series). When every crossing of a boundary has a fixed price, cross fewer times.
Processes, threads and scheduling
A process is a running program with everything that belongs to it: its own address space (previous post), open files, permissions. A thread is a line of execution inside a process: it has its own registers and stack, but shares the address space with the other threads of the same process. Measuring the cost of creating and then cleaning up each:
| Each time | |
|---|---|
| Create a thread and wait for it to finish | 13.6 µs |
fork a child process, let it exit, and wait |
267 µs |
A process costs about 20 times more: the kernel has to build a new address space (though thanks to copy-on-write from the previous post, it doesn't copy the data), new page tables, then tear it all down when the process exits. That's why the Python series recommends creating processes once (a worker pool) and reusing them, instead of creating a new one for each job.
A machine has a handful of CPU cores, but at any moment hundreds of threads exist. Deciding which thread runs on which core, and for how long, is the job of the kernel's scheduler. Each thread gets a time slice (a few milliseconds), then is paused so another can run; a thread that's waiting (reading a file, waiting on the network, waiting for a lock) isn't scheduled until what it's waiting for is done. Switching from one thread to another is called a context switch: save the old thread's registers, load the new thread's, and if it's another process, switch page tables too.
Measured with two processes passing one byte back and forth through two pipes: process A writes, process B, asleep waiting to read, is woken up, writes back, process A is woken up:
for (int i = 0; i < trips; i++) { write(to_child[1], &c, 1); read(to_parent[0], &c, 1); }
Each round trip takes 3.6 µs: four system calls, plus waking a sleeping process twice. That's about 16,000 CPU cycles to pass one byte. Every exchange between two processes (or two threads that wait on each other) pays this price, and that's why async I/O on a single thread (the Python series) handles thousands of connections more efficiently than thousands of threads.
Devices and drivers
Disks, network cards, keyboards, GPUs: each device speaks its own "language" of control registers, command formats and signals. The part of the kernel that speaks that language is the driver. Programs don't talk to devices directly; they call the kernel (or a library), and the driver handles the rest.
Two mechanisms keep devices from bothering the CPU too much:
- DMA (direct memory access): the device reads or writes RAM itself, for example a network card copying a whole packet into memory without the CPU copying it byte by byte.
- Interrupts: when it's done, the device sends a signal that makes the CPU pause what it's doing and run a piece of kernel code to handle it. That's how a thread waiting on the network gets woken up right when the data arrives.
A GPU is a device like that too. When PyTorch runs a computation on the GPU, it doesn't run it: it writes a command into a command queue, through the operating system's graphics library (Metal on this machine, CUDA with NVIDIA GPUs) and the driver, then returns immediately. The GPU takes commands off the queue and runs them later, in parallel with the CPU. The CPU only waits when the program really needs the result (torch.mps.synchronize(), or when copying the result back). Every GPU measurement below has to call that wait, otherwise it only measures how long it takes to queue the commands, not to run them.
CPU and GPU: two designs
Part 2 showed that a CPU core is a complex machine designed to get through one sequence of instructions as fast as possible: predicting branches, looking ahead hundreds of instructions to run them out of order, and several large levels of cache to hide memory latency. Most of a CPU core's area goes to that management, not to arithmetic.
A GPU goes the other way: drop nearly all that cleverness, spend the area on a great many simple arithmetic units, and bet that the work has millions of identical, independent pieces.
| CPU | GPU | |
|---|---|---|
| Cores | A few to a few dozen, each very powerful | Thousands of simple arithmetic units |
| One instruction works on | One (or a few, with SIMD) values | A group of threads at once (32 threads on NVIDIA GPUs) |
| Hides memory latency with | Big caches, prefetching, out-of-order execution | Switching to another group of threads while waiting |
An if branch |
Branch prediction, very good at it | Within a group, if threads go different ways, both branches run in turn |
| Memory | Low latency | Very high bandwidth |
| Best at | Complex logic, many branches, one job done fast | The same calculation over millions of elements |
The "branch" row is the most important difference. On a GPU, a group of threads runs the same instruction at the same moment. If half the group needs the if branch and the other half the else branch, the GPU runs the if branch (the other half sits waiting), then the else branch (the first half sits waiting). Branchy, hard-to-predict code, which a CPU handles with branch prediction, can cost twice as much or more on a GPU.
Measuring: the same matrix multiplication
Multiply two square matrices of 32-bit floats, on the CPU and on the GPU, at four sizes (best of 5 runs, waiting for the GPU to finish):
import time, torch
def timed(f, sync, n=5):
f(); sync()
best = 1e9
for _ in range(n):
t = time.perf_counter(); f(); sync(); best = min(best, time.perf_counter() - t)
return best
a, b = torch.randn(size, size), torch.randn(size, size)
ga, gb = a.to("mps"), b.to("mps")
cpu = timed(lambda: a @ b, lambda: None)
gpu = timed(lambda: ga @ gb, torch.mps.synchronize)
| Size | CPU | GPU | Faster |
|---|---|---|---|
| 16 × 16 | 0.001 ms | 0.527 ms | CPU, ~500 times |
| 256 × 256 | 0.023 ms | 0.420 ms | CPU, ~18 times |
| 1,024 × 1,024 | 1.06 ms (2.0 TFLOP/s) | 2.70 ms (0.8 TFLOP/s) | CPU, 2.6 times |
| 4,096 × 4,096 | 67.5 ms (2.0 TFLOP/s) | 38.8 ms (3.5 TFLOP/s) | GPU, 1.7 times |
The shape of the table matters more than any single number:
- For small jobs, the GPU loses badly. A 16 × 16 multiplication is only about 8,000 operations, but on the GPU it takes half a millisecond: almost all of it is the cost of sending the command through the driver, waiting for the GPU to pick it up and run it, then signalling completion. Measuring a single addition on one number: 102 µs on the GPU, against 0.7 µs on the CPU.
- The GPU only wins with enough work. At 4,096 × 4,096 (137 billion operations), the cost of sending the command becomes negligible, and the GPU's thousands of units finally get to run flat out.
- This CPU is unusually strong at matrix multiplication. 2 TFLOP/s in 32-bit floats is far more than a few ordinary CPU cores can do with SIMD instructions; the math library is probably using matrix-acceleration blocks built into the chip. On AI servers the gap runs hard the other way: a data-center GPU is tens of times faster than the integrated GPU here, and faster still with 16-bit or 8-bit numbers (part 5 of the LLM series) thanks to units dedicated to matrices.
Two more measurements:
| Time | |
|---|---|
Copy 64 MiB from RAM to the GPU (x.to("mps")) |
1.63 ms (38 GiB/s) |
| Copy 64 MiB back from the GPU | 1.93 ms |
torch.where(z > 0, z * 2, -z) over 16 million numbers: CPU / GPU |
5.2 ms / 4.6 ms |
- Data has to be moved. On this machine, the CPU and GPU share the same physical memory, and still copying 64 MiB takes more than 1.6 ms. With a discrete GPU, data has to cross the PCIe bus into the GPU's own memory, usually slower still. A program that runs well on a GPU puts its data there once and does a lot of work on it, rather than copying back and forth after every step.
- The "branchy" operation is written GPU-style.
torch.wheredoesn't branch per element: it computes bothz * 2and-zfor every element, then picks results according to the condition. That's exactly how a GPU handles anif: do both branches, keep one. Here CPU and GPU both take about 5 ms, because the limit is reading and writing 64 MiB of data, not the arithmetic; just like the memory wall in part 2.
Why AI runs on GPUs
Tying it back to the LLM series: part 2 showed that almost all of a language model is matrix multiplication, part 3 showed that generating text is limited by memory bandwidth, and part 5 showed servers gathering many users into one batch so each read of the weights does more work. All three are a GPU's strengths: the same calculation over huge blocks of data, high-bandwidth memory, and the more work per command sent, the better.
And the jobs that shouldn't go to the GPU: small calculations (sending the command costs more than the calculation), branchy, hard-to-predict logic, work that needs low latency for each small request, and anything that has to copy data back and forth constantly. A real program usually uses both: the CPU coordinates, reads data and makes decisions; the GPU handles the big blocks of arithmetic.
Summary
| Concept | What it really is |
|---|---|
| Kernel | The part of the OS running in privileged mode; programs must ask it through system calls |
| System call | 345 ns for one write, almost 900 times a function call; batch them |
| Process and thread | A process has its own address space; creating one costs ~20 times a thread |
| Scheduling | Time slices for threads; one round trip between two processes ~3.6 µs |
| Drivers, DMA, interrupts | The kernel talks to devices; devices copy into RAM themselves and signal when done |
| The GPU is a device | Commands are queued and run later; you must wait to measure correctly |
| CPU and GPU | A few clever cores and big caches, versus thousands of simple units and high bandwidth |
| Branches on a GPU | A group whose threads diverge runs both branches |
| When the GPU wins | Big, uniform jobs with the data already on the GPU; small jobs lose to the cost of sending commands |
The first four posts stayed inside one machine. The last one goes outside: what happens in the half second or so between typing viet.sarl and the page appearing, through each layer of computer networking.