Serving many users at once: batching and quantization

Part 3 concluded that when generating text, the model mostly just waits for memory: every token has to read all 1.98 GB of weights, while the processor sits idle. That post also left a consequence hanging: if you have to read 1.98 GB anyway, read it once and use it for many people at the same time. This post builds exactly that: a small "server" on top of the model written in parts 2 and 3, serving many users with a single copy of the weights.

The route: batching and what it costs, why "wait for the whole group to finish" scheduling is wasteful and how continuous batching fixes it, how the KV cache decides how many users fit in memory, then quantization: making the weights smaller by rounding them, and measuring how far the rounding can go before the model breaks.

Every number in this post was measured on a real machine, with Python 3.12, PyTorch 2.14 and transformers 5.20, on the CPU with 2 threads. The experiments are kept small (a few seconds each), so the absolute numbers differ a little from part 3; what matters is the ratios.

Batching: one read, many users

In part 3, one decode step sent one token through the model. With many users, each has their own new token, at a different position, with their own KV cache. The trick: inside the model, only attention needs to know whose token it is. Every multiplication with the weights (q, k, v, o, the MLP) handles each token independently, so the tokens of everyone can be gathered into one matrix and multiplied once:

def decode_step(next_ids: list[int], caches: list[dict]) -> torch.Tensor:
    """One decode step for many users at once: one new token each, each with its own cache and position."""
    x = W["embed_tokens.weight"][next_ids]                         # (users, 896)
    positions = [c[0][0].shape[1] for c in caches]                 # every user is at a different length
    for i in range(LAYERS):
        p = f"layers.{i}.self_attn."
        h = rms_norm(x, W[f"layers.{i}.input_layernorm.weight"])
        q = h @ W[p + "q_proj.weight"].T + W[p + "q_proj.bias"]    # one matrix multiply for all users:
        k = h @ W[p + "k_proj.weight"].T + W[p + "k_proj.bias"]    # the weights are read once
        v = h @ W[p + "v_proj.weight"].T + W[p + "v_proj.bias"]
        out = []
        for u, cache in enumerate(caches):                         # attention: each user looks at their own cache
            pos = torch.tensor([positions[u]])
            qu = rope(q[u].view(1, H, HD).transpose(0, 1), pos)
            ku = rope(k[u].view(1, KV, HD).transpose(0, 1), pos)
            vu = v[u].view(1, KV, HD).transpose(0, 1)
            ck, cv = torch.cat([cache[i][0], ku], 1), torch.cat([cache[i][1], vu], 1)
            cache[i] = (ck, cv)
            a = torch.nn.functional.scaled_dot_product_attention(qu, ck, cv, enable_gqa=True)
            out.append(a.transpose(0, 1).reshape(D))
        x = x + torch.stack(out) @ W[p + "o_proj.weight"].T
        x = x + mlp(rms_norm(x, W[f"layers.{i}.post_attention_layernorm.weight"]), i)
    return rms_norm(x, W["norm.weight"]) @ W["embed_tokens.weight"].T  # (users, 151936)

The rms_norm, rope and mlp functions are part 2's, unchanged. The condition for trusting it: two users running in one batch must get exactly the same tokens as when each runs alone. Checked with two different questions, 16 tokens each: identical.

The time for one step, as more people wait for their next token:

From 2 to 16 users, a step only gets about 40% slower, while the tokens produced per step go up 8 times. Serving 16 people at once gives 6 times the tokens per second of serving them one at a time, with the same single copy of the weights in memory. The step from 1 to 2 users alone almost doubles the time before things flatten out, probably because the math library has a separate, much faster path for multiplying by exactly one vector. The overall shape is just what part 3 predicted: the main cost is reading the weights, and it's shared by everyone in the batch.

The price: each person receives text more slowly (36.5 down to 13.7 tokens/s). Every LLM serving system has to pick a point on this curve: big batches make each token cheaper, small batches make the text flow faster for each person.

Scheduling: don't make the group wait for the slowest

Batching raises a new question: who gets into the batch, and when? The simplest way is static batching: put 4 requests in a group, run until the whole group is done, then take the next group. The problem is that answers differ in length. "What's 1 + 1?" needs a few tokens, a story needs hundreds. Within a group, people who finish early still hold their place until the last one finishes, and the requests in the queue have to wait.

Continuous batching (what vLLM and modern servers do) schedules step by step: as soon as someone finishes, their place goes to the next request in the queue, at the very next decode step.

def continuous(requests, slots=4):
    queue, active = list(requests), []
    while queue or active:
        while queue and len(active) < slots:                      # a free slot: let the next request in now
            request = queue.pop(0)
            next_id, cache = prefill(request.prompt)
            active.append([request, next_id, cache, 1])
        out = decode_step([a[1] for a in active], [a[2] for a in active]).argmax(-1).tolist()
        for a, next_id in zip(active, out):
            a[1], a[3] = next_id, a[3] + 1
        for a in [a for a in active if a[3] >= a[0].max_tokens]:
            active.remove(a)                                      # finished: its slot frees up immediately

Measured with 12 requests arriving together, each needing a different number of tokens (8 to 64, 381 in total), at most 4 users per batch:

Same model, same maximum batch of 4, only the scheduling differs: 1.65 times faster, and each person waits a third less. With static batching, the group with the 64-token request leaves the other three idle for most of the time; with continuous batching, no slot sits empty while people are still queueing.

One detail in the code above: a new request entering the batch has to prefill its prompt first. Prefill is limited by compute (part 3) while decode is limited by memory, so real servers mix them more cleverly: they cut a long prompt's prefill into chunks and run them between other people's decode steps, so one long prompt doesn't stall the whole batch.

The KV cache decides how many users fit

Batching shares the weights, but the KV cache can't be shared: each user has their own, growing with the length of the conversation. Part 3 worked it out: 12 KiB per token for Qwen2.5-0.5B, 56 KiB for the 7B version, 320 KiB for the 72B. The memory left after holding the weights is the budget for caches, and it decides how big the batch can get.

A worked example (an estimate, not a measurement, ignoring memory for temporary computations): Qwen2.5-7B in bfloat16 on a 24 GB GPU. The weights take about 15.2 GB, leaving about 8.8 GB for caches. Each 4,096-token conversation needs 56 KiB × 4,096 = 224 MiB, so about 37 conversations fit at once. If the weights are compressed to 4 bits (section below), they shrink to about 4.3 GB, caches get almost 20 GB, and the number of conversations goes up to about 84. Quantization doesn't just make each step faster; it also makes room for a bigger batch.

There's one more kind of waste: if every user gets space reserved for the maximum length (32,768 tokens) while most conversations are only a few hundred tokens long, most of the cache memory sits empty. vLLM solves this with PagedAttention: split the cache into small fixed-size pages and allocate them as the conversation grows, the way an operating system manages virtual memory.

Quantization: rounding the weights

Part 3 showed that going from 32-bit to 16-bit weights makes decode 1.7 times faster, because there are fewer bytes to read. Going further: replace each float with a small integer plus a scale shared by a group.

def quantize(w: torch.Tensor, bits: int, group: int = 32):
    """Round each group of 32 weights to 2**bits levels; keep one float scale per group."""
    g = w.reshape(-1, group)
    qmax = 2 ** (bits - 1) - 1                                  # 127 for 8 bits, 7 for 4 bits
    scale = g.abs().amax(dim=1, keepdim=True).clamp_min(1e-12) / qmax
    q = (g / scale).round().clamp(-qmax - 1, qmax).to(torch.int8)  # the small integers that get stored
    return q, scale


def dequantize(q, scale, shape):
    return (q.float() * scale).reshape(shape)

For each group of 32 consecutive weights: find the largest absolute value, divide that range evenly into 2^bits levels, and round each weight to the nearest level. At 8 bits there are 256 levels; at 4 bits only 16; at 2 bits only 4 (−2, −1, 0, 1 times the scale). What gets stored is the small integers, plus one 16-bit scale per 32 weights.

To measure quality, apply quantize then dequantize to every matrix of Qwen2.5-0.5B in the hand-written model, then measure two things: bits per byte on a 10 KB slice of Vietnamese Wikipedia (part 4's yardstick, lower is better), and the greedy answer to "Thủ đô của Việt Nam là gì?" ("What is the capital of Vietnam?"):

• 8 bits: no measurable loss. Almost 4 times smaller, quality identical to three decimal places. 256 levels are more than enough for a model's weights.
• 4 bits: a small loss. 7 times smaller, bits per byte up 10%, the answer still right. This is the level most people running models on their own machines use.
• 3 bits: collapse. Bits per byte doubles, level with xz in part 4, and the model only prints asterisks.
• 2 bits: nonsense. Worse than gzip; the model spits out random tokens from all sorts of languages.

The rounding above is the simplest kind (round to the nearest level). Methods like GPTQ and AWQ choose the rounding more cleverly, using a little sample data to learn which weights matter, and keep better quality at 4 and 3 bits. Larger models also usually tolerate quantization better than the 0.5B one here. But the general shape is this: almost free down to 8 bits, cheap at 4 bits, then a cliff.

And speed? The experiment above rounds and then converts back to 32-bit floats before multiplying, so it only measures quality; it isn't any faster. To be truly faster, the software has to read the small integers straight from memory and unpack them during the multiplication (libraries like llama.cpp and vLLM have such functions). Then, by part 3's formula (max tokens per second ≈ memory bandwidth ÷ bytes of weights), the 0.53 GB int8 version has an upper bound almost 4 times the float32 one, and the 0.28 GB int4 version about 7 times. That's an estimate from the formula, not a measurement.

Putting it together: what a real LLM server does

Every piece of the series so far shows up in an LLM server like vLLM:

From the user's side, that server looks exactly like the chatbot in the Python series: an endpoint that takes a question and streams back token by token. The difference is that behind the endpoint, dozens of people's tokens are generated in the same step, with the same copy of the weights.

Summary

Five posts have gone from tokens to the server. The last one turns back to the people building products: using a model to make a real application, with system instructions, tool calling, the agent loop, and answering questions from your own documents.