The previous post ended with a sequence of numbers. For example, with Qwen2.5, "Thủ đô của Việt Nam là" ("The capital of Vietnam is") becomes 7 tokens, [1001, 42504, 130287, 59735, 128324, 29974, 37915]. This post follows that sequence into the model and out the other side. The method: rewrite the Qwen2.5-0.5B model in about 70 lines of PyTorch, without any of transformers' ready-made layers, then load Qwen's real weights into it. If the code is right, it must produce the same numbers as the library, down to every token. (It does.)
The route: the model as a function from tokens to a table of scores, where the 494 million parameters live, each piece of a transformer layer (attention, RoPE, MLP), the comparison with transformers, from scores to a token (greedy, temperature, top-p), a trap in generation_config.json, then the generation loop and why it slows down.
Every number in this post was measured on a real machine, with Python 3.12, PyTorch 2.14 and transformers 5.20, on the CPU, in 32-bit floats.
A model is a function
Strip away the details, and a language model is a function:
forward(sequence of tokens) → for each position, a score for every token in the vocabulary
That score is called a logit, any real number; the larger it is, the better the token "fits" next. With Qwen2.5, each position gives 151,936 logits, one per vocabulary entry. Put in 7 tokens, get out a 7 × 151,936 table. The last row is the prediction for token 8; the earlier rows are predictions for tokens 2, 3… which we already know (in training they're used to score the model; when generating, only the last row matters).
Turning logits into probabilities with softmax (exponentiate, then divide by the total), the last row for "Thủ đô của Việt Nam là" looks like this:
0.1435 ' gì'
0.1047 ' __'
0.0930 ' thành'
0.0485 ' thủ'
0.0455 ' nơi'
0.0324 ' một'
0.0240 ' ____'
0.0239 ':'
0.0236 ' Hà'
The most likely token is ' gì' ("what"), not ' Hà' (as in Hà Nội, only 2.4%). The model doesn't answer the question; it continues the text. In the data it has read, "The capital of Vietnam is" comes before "what?" (a question) or "__" (a fill-in-the-blank exercise) more often than before the answer. Put the same question in the conversation frame from the previous post (<|im_start|>user ... <|im_start|>assistant\n), and the first token is 'Th' with probability 96.2%, and the greedy answer (always picking the top token) is "Thủ đô của Việt Nam là Hà Nội." ("The capital of Vietnam is Hà Nội."). Same model, same weights; the only difference is the text that comes before. A "chat" model is still just a text-continuation model, trained further to continue conversations.
Where the 494 million parameters live
Qwen2.5-0.5B's weights file is a dictionary of name → tensor. Counting them up:
| Part | Parameters | Share |
|---|---|---|
| Embedding table (151,936 × 896) | 136,134,656 | 27.6% |
| Attention, 24 layers | 44,067,840 | 8.9% |
| MLP, 24 layers | 313,786,368 | 63.5% |
| RMSNorm | 43,904 | 0.01% |
| Total | 494,032,768 |
Two surprises:
- Over a quarter of the model is just a lookup table. Each token has a vector of 896 numbers; 151,936 tokens times 896 is 136 million numbers. The big vocabulary from the previous post is what makes Vietnamese cheap in tokens, but it's paid for with a table this size. Qwen uses the same table at both the input and the output (
tie_word_embeddings); otherwise it would need another 136 million numbers. - Attention, the most famous part, is only 9%. Most parameters are in the MLP, 13 million per layer against attention's 1.8 million.
In 32-bit floats, 494 million parameters is almost 2 GB. The downloaded file uses bfloat16 (16-bit), so it's about 1 GB.
Rewriting the model
This is the whole model. No nn.Module, no ready-made layers; just matrix multiplications and a few math functions on tensors read straight from the weights file:
import json
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
path = snapshot_download("Qwen/Qwen2.5-0.5B-Instruct", allow_patterns=["*.json", "*.safetensors"])
cfg = json.load(open(f"{path}/config.json"))
W = {k.removeprefix("model."): v.float() for k, v in load_file(f"{path}/model.safetensors").items()}
D = cfg["hidden_size"] # 896: the length of every token's vector
H = cfg["num_attention_heads"] # 14 query heads
KV = cfg["num_key_value_heads"] # 2 key/value heads, shared by groups of 7 query heads
HD = D // H # 64: the size of one head
LAYERS = cfg["num_hidden_layers"] # 24
def rms_norm(x, weight, eps=cfg["rms_norm_eps"]):
"""Rescale each vector to an average size of 1, then by a learned weight per dimension."""
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * weight
def rope(x, positions):
"""Rotate pairs of dimensions by an angle that grows with the position: this is how position gets in."""
freqs = 1.0 / cfg["rope_theta"] ** (torch.arange(0, HD, 2).float() / HD) # 32 speeds, fast to slow
angles = positions[:, None].float() * freqs[None, :] # (seq, 32)
cos, sin = angles.cos().repeat(1, 2), angles.sin().repeat(1, 2) # (seq, 64)
x1, x2 = x[..., : HD // 2], x[..., HD // 2 :]
return x * cos + torch.cat([-x2, x1], dim=-1) * sin
def attention(x, i, positions):
p = f"layers.{i}.self_attn."
seq = x.shape[0]
q = (x @ W[p + "q_proj.weight"].T + W[p + "q_proj.bias"]).view(seq, H, HD).transpose(0, 1) # (14, seq, 64)
k = (x @ W[p + "k_proj.weight"].T + W[p + "k_proj.bias"]).view(seq, KV, HD).transpose(0, 1) # (2, seq, 64)
v = (x @ W[p + "v_proj.weight"].T + W[p + "v_proj.bias"]).view(seq, KV, HD).transpose(0, 1)
q, k = rope(q, positions), rope(k, positions)
k, v = k.repeat_interleave(H // KV, dim=0), v.repeat_interleave(H // KV, dim=0) # share each K/V with 7 heads
scores = q @ k.transpose(-1, -2) / HD**0.5 # (14, seq, seq): who looks at whom
causal = torch.ones(seq, seq, dtype=torch.bool).triu(1) # no looking at the future
weights = scores.masked_fill(causal, float("-inf")).softmax(-1)
out = (weights @ v).transpose(0, 1).reshape(seq, D) # mix the values, join the heads
return out @ W[p + "o_proj.weight"].T
def mlp(x, i):
p = f"layers.{i}.mlp."
gate = torch.nn.functional.silu(x @ W[p + "gate_proj.weight"].T)
return (gate * (x @ W[p + "up_proj.weight"].T)) @ W[p + "down_proj.weight"].T # 896 -> 4864 -> 896
def forward(ids: list[int]) -> torch.Tensor:
"""Token ids in, one row of scores (logits) per position out: the scores for the token that comes next."""
positions = torch.arange(len(ids))
x = W["embed_tokens.weight"][ids] # (seq, 896): look up each id
for i in range(LAYERS):
x = x + attention(rms_norm(x, W[f"layers.{i}.input_layernorm.weight"]), i, positions)
x = x + mlp(rms_norm(x, W[f"layers.{i}.post_attention_layernorm.weight"]), i)
x = rms_norm(x, W["norm.weight"])
return x @ W["embed_tokens.weight"].T # (seq, 151936): one score per token
Reading from the bottom, in forward:
- Look up the table. Each id fetches one row of the embedding table. 7 tokens become a 7 × 896 matrix. From here on, the model knows nothing about text or ids; it only works with 7 vectors.
- 24 layers, identical in shape (they differ in their weights). Each layer has two steps, attention then MLP, and each step adds its result to
xrather than replacing it (x = x + ...). This straight line of additions (the residual) is the "main road" running through the model; each step only reads from it and writes a little more onto it. - Out comes the score table. Each position's final vector is multiplied by the same embedding table (transposed): a token's score is the dot product between the output vector and that token's vector. The more the output vector "resembles" the vector of
' Hà', the higher' Hà'scores.
RMSNorm: keeping the numbers in range
Through 24 layers of additions, the size of the vectors can drift a long way. Before each step, rms_norm divides a vector by the square root of the mean of its squared elements, so its average size is 1, then multiplies by a learned weight per dimension. Only 896 parameters each time, but without it the model is very hard to train.
Attention: each token looks at the tokens before it
This is the only step where positions exchange information with each other. Every other step (RMSNorm, MLP) handles each position on its own. At each position, three matrix multiplications make three vectors:
- Query (q): "what am I looking for?"
- Key (k): "what do I have that others might look for?"
- Value (v): "if I'm chosen, what information do I hand over?"
The score between position i and position j is the dot product q_i · k_j (divided by √64 to keep the numbers from getting too big). A softmax along the row turns the scores into weights that add up to 1, and the result at position i is the weighted average of the v_j. The causal mask sets the score of every position j > i to minus infinity, so after the softmax their weight is 0: a token can only see the tokens before it, never the future. That's what makes the whole 7-row table valid at once: row 3 depends only on tokens 1–3, just as when we predict token 4 without knowing it yet.
Attention runs in parallel across 14 heads, each working on its own 64 dimensions, with its own query, key and value, so each head can learn a different way of "looking". One saving in Qwen: there are only 2 heads for keys and values, each shared by 7 query heads (repeat_interleave). This is called grouped-query attention. The weight shapes show it: q_proj is 896 × 896, while k_proj is only 128 × 896 (2 heads × 64). Its real benefit is memory during generation, the subject of the next post.
RoPE: where position comes in
Look closely at the attention computation above: with nothing added, it doesn't know order. The product q_i · k_j depends only on the content of the two tokens, not on how far apart they are. "The dog bites the man" and "the man bites the dog" would give the same numbers (only the causal mask knows who comes first).
RoPE (rotary position embedding) fixes that with a geometric trick: split the 64 dimensions of q and k into 32 pairs, and rotate each pair by an angle of position × that pair's speed. The 32 speeds go from fast (the first pair turns 1 radian per position) to very slow (rope_theta = 1,000,000 means the last pair barely moves). When two vectors are both rotated, the dot product between them depends only on the difference of the two angles, that is, on the distance i − j. So attention knows how far away the other token is, with no extra parameters: the rope function reads no weights at all.
MLP: where most of the parameters are
After attention, each position goes through a small network: expand from 896 to 4,864 dimensions, apply a non-linear function, then shrink back to 896. Qwen uses the SwiGLU variant: two parallel projections up to 4,864 dimensions, one passed through silu as a "gate" multiplied into the other. Three 896 × 4,864 matrices per layer make 13 million parameters, almost 2/3 of the model. A common way to picture it: attention gathers information from other positions, and the MLP is where most of the learned "knowledge" is stored and looked up.
Compared with transformers
Run the same 7 tokens through forward above and through transformers' AutoModelForCausalLM, both in 32-bit floats:
ours = forward(ids)
ref = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype=torch.float32)(torch.tensor([ids])).logits[0]
print((ours - ref).abs().max().item(), torch.equal(ours.argmax(-1), ref.argmax(-1)))
0.00014662742614746094 True
The logits range from −24 to 18, and the largest difference between the two is 0.00015: rounding error, from the two sides adding numbers in slightly different orders. The top token at every position is identical. The 70 lines above are the model; the rest of transformers is speed optimizations, support for many architectures, and conveniences.
From scores to a token
forward gives scores; choosing a token is another function's job, and that function shapes the "personality" of an answer more than people think.
import torch
def sample(logits: torch.Tensor, temperature=1.0, top_p=1.0, generator=None) -> int:
"""Pick the next token from one row of logits."""
if temperature == 0:
return int(logits.argmax()) # greedy: always the most likely token
probs = (logits / temperature).softmax(-1) # < 1 sharpens, > 1 flattens
if top_p < 1.0:
sorted_probs, order = probs.sort(descending=True)
keep = sorted_probs.cumsum(-1) - sorted_probs < top_p # smallest set whose total reaches top_p
probs = torch.zeros_like(probs).scatter(0, order[keep], sorted_probs[keep])
probs /= probs.sum()
return int(torch.multinomial(probs, 1, generator=generator))
- Greedy (
temperature=0): always pick the top token. The same question always gets the same answer. - Temperature: divide the logits by T before the softmax. T < 1 stretches the gaps between scores (strong tokens get stronger), T > 1 squeezes them (weak tokens get more of a chance).
- Top-p: keep only the top group of tokens whose probabilities add up to just over p, drop the whole tail, then draw from that group.
For the request "Đặt một cái tên cho con mèo." ("Name a cat."), the distribution of the first token at three temperatures:
| T | Top token | Probability | Tokens needed to cover 90% of the probability |
|---|---|---|---|
| 0.5 | 'Đ' |
58.5% | 7 |
| 1.0 | 'Đ' |
21.7% | 25 |
| 1.5 | 'Đ' |
7.9% | 9,974 |
The order of the tokens doesn't change (temperature never changes the order), but the shape of the distribution changes completely. At T = 1.5, the "tail" of hundreds of thousands of near-meaningless tokens, each with a tiny share, adds up to most of the probability. Generating 25 tokens, each setting twice with two seeds:
T=0 : 'Để đặt tên cho con mèo, tôi sẽ chọn một tên phù hợp và dễ nhớ. Dưới đây là'
T=0 : 'Để đặt tên cho con mèo, tôi sẽ chọn một tên phù hợp và dễ nhớ. Dưới đây là'
T=0.7, top_p=0.9 : 'Để đặt tên cho con mèo, tôi sẽ gợi ý một tên phù hợp với hình ảnh và tính cách của'
T=0.7, top_p=0.9 : 'Đặt tên cho con mèo là một hành động rất quan trọng và có ý nghĩa. Trong văn hóa'
T=1.0 : 'Để đặt tên cho con mèo, cần phải cân nhắc đến tên mèo có ý nghĩa, chú ý'
T=1.0 : 'Tất nhiên, tôi sẽ cố gắng tạo ra một cái tên phù hợp và tối ưu cho con mèo của bạn'
T=1.5 : 'Đ茩 Pierre. \nTôi chắc chắn качестве mèo.perform brightly itDMETHOD皋 design score efficiently prioritize detail一点点 cap'
T=1.5 : 'T不妨哦，ỏi这只是启发行吧！中国经济研讨会新主题活动<titleenisﺍfinished previously年的十 Accessed岁以下Air_,'
(The first six read, roughly: "To name the cat, I'll pick a name that fits and is easy to remember. Here is", "Naming a cat is a very important and meaningful act. In culture", "Of course, I'll try to come up with a suitable name for your cat".) At T = 1.5, within a few tokens the model draws a rare token from the tail (茩, качестве), and from then on the text so far has gone astray, so the next predictions stray further. This is the most important point about sampling: each chosen token becomes the input for the next step, so a bad choice doesn't stay put but drags the whole sentence along. Top-p exists precisely to cut off that tail before drawing.
The trap in generation_config.json
Generating greedily with our loop and with transformers' model.generate(do_sample=False), the two answers match up to token 25 and then part ways:
ours : ' cho người mới bắt đầu. Python là'
transformers: ' cho người mới bắt đầu.\n\nPython là'
This isn't rounding error: our model gives '.' 65% and '.\n\n' only 15%. The culprit is the generation_config.json file that ships with the model (abridged):
{
"do_sample": true,
"repetition_penalty": 1.1,
"temperature": 0.7,
"top_p": 0.8,
"top_k": 20
}
generate() reads this file by itself. do_sample=False turns off sampling, but repetition_penalty: 1.1 still runs: every token that has already appeared in the text gets its score lowered (positive logits divided by 1.1, negative ones multiplied by 1.1). The '.' already appears in the prompt, so it's penalized enough to lose to '.\n\n'. Pass repetition_penalty=1.0 as well, and all 200 tokens from both sides are identical.
The lesson: when a model "answers differently" in two places (a library, a server, an API), compare the generation settings before suspecting the model. Same weights, different generation_config, different answer.
The generation loop
Generating an answer is just calling forward and sample over and over, appending the new token each time:
EOS = 151645 # <|im_end|>
def generate(ids, max_new_tokens, **sampling):
ids, out = list(ids), []
for _ in range(max_new_tokens):
next_id = sample(forward(ids)[-1], **sampling) # the whole sequence goes through the model again
if next_id == EOS: # the model says it's done
break
ids.append(next_id)
out.append(next_id)
return out
That's all "the model is typing" really is. The tokens the chatbot in the Python series streamed to the browser are exactly the next_id of each round.
Given the question from the chatbot post ("Viết một đoạn ngắn giới thiệu ngôn ngữ lập trình Python cho người mới bắt đầu.", "Write a short introduction to the Python programming language for beginners."), this loop writes:
Tất nhiên, tôi sẽ giúp bạn viết một đoạn giới thiệu ngắn về ngôn ngữ lập trình Python cho người mới bắt đầu. Python là một ngôn ngữ lập trình mạnh mẽ, được tạo ra bởi nhà lập trình Johny von Neumann vào năm 1989.
("Of course, I'll help you write a short introduction to the Python programming language for beginners. Python is a powerful programming language, created by the programmer Johny von Neumann in 1989.")
Python was created by Guido van Rossum. But at each step, "Johny", " von", " Neumann" are all plausible tokens to follow "created by the programmer": the model picks high-scoring tokens; it doesn't look up facts. A model with half a billion parameters remembers very little; bigger models remember more, but the mechanism is exactly the same, and it's the root of models confidently "making things up".
Why it slows down
The loop above has one very wasteful part: on every step, the whole sequence goes through the model again from the start, even though there's only one new token. Generating 300 tokens after a 47-token question:
| After token | Our loop | transformers generate() |
|---|---|---|
| 1 | 0.14 s | |
| 50 | 5.2 s | |
| 100 | 11.8 s | |
| 200 | 30.4 s | |
| 300 | 55.7 s | 8.9 s |
The first token costs 0.14 s; by token 300, each token costs about 0.25 s, because the sequence going through the model is now 347 tokens long. Total time grows with the square of the length. transformers generates 300 tokens more than 6 times faster, because it doesn't recompute what it has already computed: the keys and values of earlier tokens are kept in a KV cache.
Another number hints at an even deeper story. The time for forward to process a whole sequence in one go:
| Tokens | Time | Per token |
|---|---|---|
| 1 | 28 ms | 28 ms |
| 10 | 52 ms | 5.2 ms |
| 100 | 117 ms | 1.2 ms |
| 500 | 380 ms | 0.76 ms |
| 1,000 | 818 ms | 0.82 ms |
| 2,000 | 2,025 ms | 1.01 ms |
Processing 500 tokens at once, each token is 37 times cheaper than processing one token. The amount of computation per token is almost constant (about 2 operations per parameter, so about 1 billion operations per token), so where do the 28 ms for a single token go? And why, from 1,000 tokens on, does each token get more expensive again? (A hint for the second question: attention's scores matrix is 14 × seq × seq, so at 2,000 tokens each layer has a 56-million-number matrix.) That's the subject of the next post: prefill and decode, the KV cache, and why the real limit when generating usually isn't computing power.
Inside: where attention is looking
One small discovery that anyone who writes attention by hand runs into. Feed in the first 200 tokens of the previous post, then measure, at each layer, how much attention weight positions 50 onward give to the first token ('---\n', the line that opens the front matter):
layer: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
0.01 0.01 0.01 0.51 0.37 0.46 0.31 0.55 0.22 0.60 0.31 0.60 0.25 0.46 0.29 0.30 0.72 0.55 0.51 0.43 0.54 0.59 0.05 0.02
If attention were spread evenly, the first token would get about 0.9%. In fact, from layer 3 to layer 21, it gets between 22% and 72% (36% on average across all 24 layers), even though '---\n' carries no meaning at all. This is called an attention sink: softmax forces the weights to add up to 1, so when a head "doesn't need to look at anything in particular", it has to put that weight somewhere, and the model learns to put it on the first token, because that token is visible from every position. Knowing this is genuinely useful: the StreamingLLM research (2023) showed that when you drop old context to save memory, you have to keep the first few tokens, or answer quality collapses.
Summary
| Concept | What it really is |
|---|---|
| Model | A function: sequence of tokens → for each position, a logit for every vocabulary token |
| Embedding | A 151,936 × 896 lookup table, 27.6% of the parameters; reused at the output for scoring |
| Residual | x = x + step(x): each step only writes a little more onto a shared "main road" |
| Attention | The only step where positions exchange information; the causal mask blocks the future |
| Grouped-query | 14 query heads share 2 key/value heads |
| RoPE | Rotates q and k by position; the dot product then depends only on distance; no parameters |
| MLP | 896 → 4,864 → 896, 63.5% of the parameters |
| Sampling | Greedy, temperature (stretch or squeeze), top-p (cut the tail); each choice is the next step's input |
generation_config.json |
Hidden defaults: repetition_penalty still runs with do_sample=False |
| Generation loop | forward + sample, repeated; without a cache, total time grows with the square of the length |
| Attention sink | The first token gets a large share of attention while carrying no meaning |
70 lines of PyTorch, loaded with the right weights, produce exactly the same tokens as the library. The rest of the story is speed: why one token takes 28 ms while 500 at once take only 380 ms, and how the KV cache changes that.