The previous three posts treated the 494 million numbers in Qwen2.5's weights file as a given: part 2 loaded them into 70 lines of PyTorch, part 3 measured how long reading them takes. This post answers the remaining question: where do those numbers come from?
The method: take exactly the pieces of Qwen from part 2 (embedding, attention with RoPE, MLP, RMSNorm), shrink them down to 3.2 million parameters, initialize them with random numbers, and train on Vietnamese Wikipedia. All of it runs on a CPU, using 2 threads, in 58 seconds. The result can't write good prose, but it's enough to see every stage: from a jumble of meaningless characters, to common words, to grammatical phrases.
The route: the goal of training, the data (and why 98.5% of Vietnamese Wikipedia gets thrown away), the tokenizer, the model, the training loop, watching text take shape step by step, measuring the model as a compression algorithm, then comparing with Qwen.
Every number in this post was measured on a real machine, with Python 3.12, PyTorch 2.14 and tokenizers 0.23, on the CPU. The experiment is deliberately small so it runs lightly on an ordinary laptop.
The goal: be less surprised
Part 2 showed that a model is a function: tokens in, a probability distribution over the next token out at each position. Training has a single goal: make the probability the model gives to the token that really comes next as high as possible.
The number that measures this is called the loss (cross-entropy): take the probability the model gave the correct token, and compute its −log. Give the correct token 100% and the loss is 0; give it 1% and the loss is 4.6. You can think of the loss as the model's surprise at seeing the real token. Averaged over every position of every passage, it's a single number to make smaller.
A model that knows nothing spreads probability evenly over the whole vocabulary. With 8,192 tokens, each gets 1/8,192, so the loss is ln(8192) = 9.01. This post's random model, before its first training step, has a measured loss of 9.04: it really knows nothing.
Nobody labels anything here. The text is its own answer key: at each position, the "right answer" is the token right after it. That's why language models can be trained on any text at all, and the more text the better.
The data
Wikimedia publishes Vietnamese Wikipedia on Hugging Face as parquet files. Two of the four files (about 120 MB) hold 644,340 pages. Opening a few at random:
Pleurostelma schimperi là một loài thực vật có hoa trong họ La bố ma. Loài này được (Vatke) Liede
miêu tả khoa học đầu tiên năm 1994.
Senecio platanifolius là một loài thực vật có hoa trong họ Cúc. Loài này được Benth. miêu tả khoa
học đầu tiên năm 1840.
("Pleurostelma schimperi is a species of flowering plant in the family Apocynaceae. It was first scientifically described by (Vatke) Liede in 1994.") Vietnamese Wikipedia has a huge number of stub articles generated by bots from the same template, mostly about species. In one file, the median page is only 193 characters long. Trained on this, the model would become very good at exactly one sentence: "… is a species of flowering plant in the family …". So before training, the data is filtered: only articles of 3,000 characters or more that don't open with "là một loài" ("is a species") are kept.
| Articles | |
|---|---|
| The two original files | 644,340 |
| After filtering | 9,578 (1.5%) |
98.5% of the pages are thrown away, but what's left is still 105 MB of real text: biographies, history, geography, music. For real models, filtering data is a large part of the work, and data quality often matters more than quantity. 2% of the articles (191) are set aside as test data, never used for training.
The tokenizer
The model needs a vocabulary. Instead of Qwen's 151,665 tokens (for a small model, the embedding table would be many times bigger than everything else), we learn our own 8,192 tokens with exactly the BPE algorithm from part 1, this time using the Rust implementation in the tokenizers library instead of the hand-written Python one:
from tokenizers import Tokenizer, decoders, models, normalizers, pre_tokenizers, trainers
tokenizer = Tokenizer(models.BPE())
tokenizer.normalizer = normalizers.NFC() # the lesson from part 1
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tokenizer.decoder = decoders.ByteLevel()
trainer = trainers.BpeTrainer(
vocab_size=8192, special_tokens=["<|endoftext|>"], initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train(["train.txt"], trainer)
Learning from about 100 MB of text takes 33 seconds. The familiar sentence "Tất nhiên, tôi sẽ giúp bạn" becomes 7 tokens, one per word, better even than Qwen (8 tokens): a small vocabulary that only learns Vietnamese spends all its room on Vietnamese. The whole corpus becomes 23.4 million tokens for training and 461,000 for testing, 4.39 bytes per token on average. Between two articles sits a <|endoftext|> token so the model knows where one ends.
The model: Qwen, shrunk
The model uses exactly the pieces from part 2, except that they're now nn.Module layers with learnable parameters, and attention uses scaled_dot_product_attention as in part 3:
| Qwen2.5-0.5B | This post's model | |
|---|---|---|
| Vocabulary | 151,936 | 8,192 |
| Vector size | 896 | 192 |
| Layers | 24 | 4 |
| Query / key-value heads | 14 / 2 | 4 / 2 |
| MLP | 4,864 | 512 |
| Maximum context | 32,768 | 128 |
| Parameters | 494 million | 3.2 million |
import torch
import torch.nn as nn
import torch.nn.functional as F
VOCAB, D, LAYERS, H, KV, FF, CTX = 8192, 192, 4, 4, 2, 512, 128
HD = D // H
class Block(nn.Module):
def __init__(self):
super().__init__()
self.norm1, self.norm2 = nn.RMSNorm(D), nn.RMSNorm(D)
self.q, self.k, self.v = nn.Linear(D, H * HD), nn.Linear(D, KV * HD), nn.Linear(D, KV * HD)
self.o = nn.Linear(H * HD, D, bias=False)
self.gate, self.up = nn.Linear(D, FF, bias=False), nn.Linear(D, FF, bias=False)
self.down = nn.Linear(FF, D, bias=False)
def forward(self, x, cos, sin):
b, t, _ = x.shape
h = self.norm1(x)
q = self.q(h).view(b, t, H, HD).transpose(1, 2)
k = self.k(h).view(b, t, KV, HD).transpose(1, 2)
v = self.v(h).view(b, t, KV, HD).transpose(1, 2)
q, k = rotate(q, cos, sin), rotate(k, cos, sin)
a = F.scaled_dot_product_attention(q, k, v, is_causal=True, enable_gqa=True)
x = x + self.o(a.transpose(1, 2).reshape(b, t, D))
h = self.norm2(x)
return x + self.down(F.silu(self.gate(h)) * self.up(h))
def rotate(x, cos, sin):
x1, x2 = x[..., : HD // 2], x[..., HD // 2 :]
return x * cos + torch.cat([-x2, x1], dim=-1) * sin
class GPT(nn.Module):
"""The same pieces as Qwen2.5 in the previous posts, just small, and with weights that start random."""
def __init__(self):
super().__init__()
self.embed = nn.Embedding(VOCAB, D)
self.blocks = nn.ModuleList(Block() for _ in range(LAYERS))
self.norm = nn.RMSNorm(D)
freqs = 1.0 / 10000 ** (torch.arange(0, HD, 2).float() / HD)
angles = torch.arange(CTX)[:, None] * freqs[None, :]
self.register_buffer("cos", angles.cos().repeat(1, 2), persistent=False)
self.register_buffer("sin", angles.sin().repeat(1, 2), persistent=False)
for p in self.parameters():
if p.dim() == 2:
nn.init.normal_(p, std=0.02)
def forward(self, ids):
t = ids.shape[1]
x = self.embed(ids)
for block in self.blocks:
x = block(x, self.cos[:t], self.sin[:t])
return self.norm(x) @ self.embed.weight.T # tied, like Qwen
Two small differences from part 2: everything has an extra batch dimension (b), because training processes many passages at once; and every matrix starts as small random numbers (a normal distribution with standard deviation 0.02). In this small model, the embedding table (8,192 × 192 = 1.57 million numbers) is almost half the parameters; in Qwen it's 27.6%.
The training loop
def batches(path, batch_size, seed):
data = np.fromfile(path, dtype=np.uint16)
rng = np.random.default_rng(seed)
while True:
starts = rng.integers(0, len(data) - CTX - 1, batch_size)
chunk = torch.from_numpy(np.stack([data[s : s + CTX + 1] for s in starts]).astype(np.int64))
yield chunk[:, :-1], chunk[:, 1:] # the target at each position is simply the next token
torch.set_num_threads(2) # keep the machine responsive
model = GPT()
opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=0.1, betas=(0.9, 0.95))
train = batches("train.bin", 16, seed=1)
for step in range(800):
for group in opt.param_groups:
group["lr"] = lr_at(step) # warm up for 50 steps, then decay along a cosine
x, y = next(train)
loss = F.cross_entropy(model(x).flatten(0, 1), y.flatten()) # how surprised the model is by the real next token
opt.zero_grad()
loss.backward() # which way to nudge each of the 3.2M numbers
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step() # nudge them
Each step:
- Take a batch: 16 passages, each 129 consecutive tokens from a random place in the corpus.
xis the first 128 tokens,ythe 128 after them, shifted by one. Each passage gives 128 prediction exercises at once, thanks to the causal mask from part 2: position 5 only sees tokens 1–5 and must guess token 6. - Compute the loss over all 16 × 128 = 2,048 positions.
loss.backward(): PyTorch walks back through every computation it did (this is backpropagation), and for each of the 3.2 million parameters computes a gradient: if this number goes up a little, does the loss go up or down, and by how much.opt.step(): move each parameter a small step in the direction that lowers the loss. AdamW is a smarter way of moving than plain "go against the gradient": it remembers the direction of previous steps to move more smoothly, and each parameter gets its own step size.
Two details keep things stable: clip_grad_norm_ trims the gradient when it's too large (common in the first steps), and the learning rate (step size) ramps up over the first 50 steps, then decays along a cosine to the end. Nothing in this loop says anything about grammar, spelling or history. Everything the model learns comes from repeating those four steps 800 times.
Watching text take shape
800 steps of 2,048 tokens each is 1.6 million tokens in total, about 7% of the corpus. The whole thing takes 58 seconds. The loss on the test data (articles the model has never seen):
| Step | Test loss | Time |
|---|---|---|
| 0 | 9.04 | 0 s |
| 50 | 7.46 | 4 s |
| 100 | 7.14 | 8 s |
| 200 | 6.76 | 15 s |
| 400 | 6.25 | 29 s |
| 600 | 5.97 | 44 s |
| 800 | 5.84 | 58 s |
The numbers are hard to feel; the text isn't. At each checkpoint, the model continues "Hà Nội là" ("Hà Nội is") and "Năm 1945," ("In 1945,") for 30 tokens (temperature 0.8, top-p 0.9, same seed, the sample function from part 2):
Step 0, random weights:
Hà Nội là Mil lỗ dõiĐánhócTAs 978trước Mạng Bàn Div Bếnect vượt vinhр tịínhchen Jocqué cắt 11oxombus...
Năm 1945, Mil khángChủĐánhóc�As 978 xứnging Bàn 1907 Bếnect vượt chở CoulcínhchenBiến cắt 11oxombus...
Any tokens from the vocabulary, stuck together with no pattern. Notice that the two lines are almost identical: a random model barely pays attention to its input.
Step 50 (4 seconds):
Hà Nội là Hưng,. tại giúpT đã động lãnh con. vận và được điều\n nhân khác thứ với nhưng Quốc 11, lấy có là một, không
Năm 1945, Hưng,. tại giúpT đã động lãnh con. vận và được điều\n nhân khác thứ với nhưng Quốc 11, lấy có là một, không
All real Vietnamese words, and all of them the most common words: "và" (and), "được" (is/was, passive), "của" (of), "tại" (at), "là một" (is a). The first thing the model learns is the simplest statistic: which words appear most. The two lines are still identical: it can't use context yet.
Step 400 (29 seconds):
Hà Nội là Hưng Sùng hậu tại vùng tại châu Tân. Ông là nhân hát của điều hợp nhân khác tấn vực nhưng cắt đi...
Năm 1945, Hưng, Có tại vùng Kỳ đã nên viết con trường vận động của điều thống, khác ta với nhưng cắt đi...
The two lines have split apart: the model has started to depend on its input. There are small grammatical phrases ("Ông là", he is; "tại vùng", in the region; "vận động", campaign), sentences with full stops, and capital letters after a full stop.
Step 800 (58 seconds):
Hà Nội là phường Đông.\n\nTơn động lãnh lập\nBèo tử\n\nNăm 16%\nĐuy Mông lấy sự quân Phương hội phong
Năm 1945, Hưng Hòa.\n\nTùi Đình Hòa, Thiên Nhĩ Kim được điều hợp nhân khác của Quân Dương Quốc, Chu lấy lãnh thành một cuộc phong
"Hà Nội là phường Đông." ("Hà Nội is Đông ward.") After "Hà Nội", the model thinks of an administrative unit. There are blank lines between paragraphs and short lines that look like Wikipedia section headings. There are things that look like Vietnamese names ("Đình Hòa", "Hưng Hòa"). Still no meaning, but it already has the shape of a Vietnamese Wikipedia article.
That order, common words first, local grammar next, document structure after that, meaning and knowledge last, is typical of training language models. Real models go through the same stages, just much, much further.
Measuring in bits: a language model is a compression algorithm
What does a loss of 5.84 mean? There's a way to turn it into an easy-to-compare number: bits per byte of text.
A model that gives probability p to the next token can be used to compress text: an arithmetic coder needs exactly −log₂ p bits to write that token. So the model's loss, converted from base e to base 2 and divided by bytes per token, is the compressed size. A model that guesses better compresses smaller. With a loss of 5.84 and 4.39 bytes per token: 5.84 / ln 2 / 4.39 = 1.92 bits per byte.
This unit doesn't depend on the tokenizer, so it can be compared with anything, including familiar compression programs. Over all the test data (2 MB):
| Bits per byte | |
|---|---|
| gzip -9 | 2.54 |
| xz -9 | 2.00 |
| This post's model, after 58 seconds | 1.92 |
And on a 20,000-character slice of the test data, to make room for Qwen2.5-0.5B (with Qwen, each 512-token chunk has to go through the model, so only a slice is measured):
| Bits per byte | |
|---|---|
| gzip -9 | 2.97 |
| xz -9 | 2.81 |
| This post's model | 1.79 |
| Qwen2.5-0.5B | 0.99 |
After one minute of training, a 3-million-parameter model already compresses Vietnamese better than xz. Qwen does almost twice as well again. (To be fair: Wikipedia is almost certainly in Qwen's training data, so it may well have read these very articles. Qwen's number here is an optimistic upper bound.) But the main point stands: "predict the next token" and "compress text" are the same problem, and a model that compresses Vietnamese well has to have grasped a lot about Vietnamese.
Compared with the real Qwen
There's a well-known estimate for the compute needed to train: about 6 operations per parameter, per token (2 for the forward pass as in part 2, about 4 for the backward pass).
| Parameters | Training tokens | Operations (6 × N × D) | |
|---|---|---|---|
| This post's model | 3.2 million | 1.6 million | ~3 × 10¹³ |
| Qwen2.5-0.5B | 494 million | up to 18 trillion (per the Qwen2.5 technical report) | ~5 × 10²² |
Qwen costs about 1.7 billion times more. At the speed of this experiment, that would be over 3,000 years; in practice, models like this are trained on many GPUs running in parallel. The gap between "Hà Nội là phường Đông" and "Thủ đô của Việt Nam là Hà Nội" ("The capital of Vietnam is Hà Nội") is that gap: the same architecture, the same four-step loop, different in model scale, data scale and data quality.
There's one more step this post's model doesn't have. Training as above (called pretraining) produces a model that continues text, like the version of Qwen that answered "Thủ đô của Việt Nam là" with ' gì' ("what") in part 2. To become an assistant, it's trained further on sample conversations framed with <|im_start|>user … <|im_start|>assistant (called supervised fine-tuning), then usually goes through a stage of learning from human ratings, or automatic rewards, of which answers are better. The loop is still forward, loss, backward, step; only the data and the scoring differ.
Summary
| Concept | What it really is |
|---|---|
| Training goal | Raise the probability of the token that really comes next; the text is its own answer key |
| Loss | −log of the correct token's probability; a random model's loss is ln(vocabulary size) = 9.01 |
| Data | Filter hard: 98.5% of Vietnamese Wikipedia pages are dropped as stubs |
| Backpropagation | loss.backward(): for each parameter, how the loss changes if it goes up |
| AdamW | Moves each parameter a small step towards lower loss, with momentum and its own step size |
| The stages | Common words → local grammar → document structure → meaning and knowledge |
| Bits per byte | Loss in bits divided by bytes per token; a language model is a compression algorithm |
| Scale | 6 × parameters × tokens; Qwen2.5-0.5B costs about 1.7 billion times this experiment |
| Pretraining and fine-tuning | Continue text first, learn to be an assistant after; the same loop |
We've now covered a model's whole life: training produces the weights, text becomes tokens, tokens go through the model, and why it's fast or slow. The next post turns to operations: when hundreds of people ask at once, how does a server serve them all without hundreds of copies of the model?