The last post of the Python series, One piece at a time, built a chatbot that answers token by token. In that post the model was a black box: text came out of model.generate() and all we did was pass it on to the browser. This series opens the box, starting with the simplest part, which is also the most often misunderstood: the model doesn't read text. It reads a sequence of integers, each called a token. The sentence "Tất nhiên, tôi sẽ giúp bạn" ("Of course, I'll help you" in Vietnamese) is 8 tokens for Qwen but 16 for GPT-4. Same sentence, twice the money, twice the time.
The route: what the model actually receives, how Vietnamese text becomes bytes (and why two strings that look identical are different), three ways to cut text, a BPE tokenizer in about 50 lines, what real tokenizers add on top, how many tokens Vietnamese costs with four popular tokenizers, then the consequences: price, context length, numbers, counting letters, and the streaming from the previous post.
Every number in this post was measured on a real machine, with Python 3.12, tiktoken 0.14, tokenizers 0.23, transformers 5.20. Tokenizers need no GPU and no model; every experiment here runs on a laptop with just pip install tiktoken tokenizers.
What the model actually receives
Send "Xin chào" ("Hello") to Qwen2.5 (the chatbot post's model), and this is what goes into the model:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
ids = tokenizer.apply_chat_template([{"role": "user", "content": "Xin chào"}], add_generation_prompt=True)["input_ids"]
print(len(ids), ids)
print([tokenizer.decode([i]) for i in ids])
32 [151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 55, 258, 139548, 151645, 198, 151644, 77091, 198]
['<|im_start|>', 'system', '\n', 'You', ' are', ' Q', 'wen', ',', ' created', ' by', ' Alibaba', ' Cloud', '.', ' You', ' are', ' a', ' helpful', ' assistant', '.', '<|im_end|>', '\n', '<|im_start|>', 'user', '\n', 'X', 'in', ' chào', '<|im_end|>', '\n', '<|im_start|>', 'assistant', '\n']
Three things stand out:
- The two words "Xin chào" are only 3 of the 32 tokens. The rest is the conversation "frame" the library adds by itself: a default system instruction, and the special tokens
<|im_start|>and<|im_end|>that mark who is speaking. The model has no notion of "messages"; it sees one continuous sequence of numbers and has learned that after<|im_start|>assistant\nit's its turn to write. - Tokens aren't words. "Xin" is cut into
X+in, while " chào" (with its leading space) is a single token. - A space belongs to the token after it.
' chào',' are',' Cloud': most modern tokenizers attach the space to the start of the next word. That's why the chatbot's answer arrived as'Tất ','nhiên, '… (the space at the end is the streamer grouping text, see the streaming section below).
Each number is an index into a table called the vocabulary. Qwen2.5's has 151,665 entries, GPT-4o's 200,019. Inside the model, each index is looked up as a vector, and every computation after that works on those vectors; the next post goes into that part. This post handles the question before it: how is that sequence of numbers made?
Text is bytes: Vietnamese in UTF-8
Before there are tokens, text has to become bytes. Nearly every modern tokenizer works on UTF-8 bytes, so understanding the UTF-8 of Vietnamese is half the story.
| Letter | UTF-8 bytes | Bytes |
|---|---|---|
| a, e, o, u | 1 | 61, … |
| ă, â, ê, ô, ơ, ư, đ, á, à, ã, é… | 2 | ê = c3 aa, đ = c4 91, à = c3 a0 |
| ạ, ả, ẽ, ệ, ế, ữ, ỹ… | 3 | ẽ = e1 ba bd, ệ = e1 bb 87 |
Letters that already existed in the old European code pages (á, à, ã, ê…) take only 2 bytes. The marks that almost only Vietnamese uses (the dot below, the hook above, and every letter with two marks such as ệ, ế, ữ) live in the Unicode block "Latin Extended Additional", from U+1EA0 to U+1EF9, and every letter in that range starts with e1 ba or e1 bb. Remember these two byte pairs; in a moment they'll be the first thing a tokenizer learns on its own.
All the Vietnamese posts on this site add up to 47,974 characters but 60,011 bytes; the English versions to 51,808 characters and 51,913 bytes. English is almost one byte per character, Vietnamese 1.25 on average.
Two strings that look identical
Unicode lets you write ệ two ways:
- Precomposed (NFC): one character,
U+1EC7, 3 bytes. - Decomposed (NFD): the letter
e+ a combining dot belowU+0323+ a combining circumflexU+0302, three characters, 5 bytes.
import unicodedata
s = "Tiếng Việt"
nfd = unicodedata.normalize("NFD", s)
print(len(s), len(s.encode()), "|", len(nfd), len(nfd.encode()), "| equal:", s == nfd)
10 14 | 14 18 | equal: False
On screen the two strings are identical, but to Python (and to a tokenizer) they're different strings. Decomposed text isn't rare: Unikey, the most common Vietnamese keyboard tool, has a "Unicode tổ hợp" (decomposed Unicode) mode, and some PDF files and file names use it too. The measurements below show what it costs: with most tokenizers, the same post in NFD costs twice the tokens.
Three ways to cut text
There are three obvious ways to turn text into a sequence of numbers, and all three have problems:
| Cut by | Vocabulary | Sequence length | Problem |
|---|---|---|---|
| One token per byte | 256 | Longest: "Tiếng Việt" is 14 | The model has to work to reassemble each letter; long sequences are slow and use memory |
| One token per character | ~150,000 Unicode characters | Still long: 10 | Most characters almost never appear |
| One token per word | Unbounded | Short: 2 | Unknown words, names and typos are a dead end |
The longer the sequence, the slower the model: each token goes through the whole model, and the previous post saw a 180-token answer take over 10 seconds on a CPU. The bigger the vocabulary, the heavier the vector table and the less each entry gets trained. The answer almost every model uses today sits in between: subwords, pieces smaller than words but bigger than bytes. Common words become one token, rare words are assembled from a few pieces, and in the worst case there are always bytes to fall back on, so no text is ever "impossible to write".
The most common algorithm for finding those pieces is BPE (Byte Pair Encoding).
Writing a BPE tokenizer
BPE fits in one sentence: start from bytes, then keep merging the most frequent pair of neighbouring tokens into a new token. After a few thousand merges, common letters, syllables and words form on their own.
import re
from collections import Counter
SPLIT = re.compile(r" ?\w+| ?[^\w\s]+|\s+") # a word with its leading space, punctuation, whitespace
def train(text: str, num_merges: int):
"""Learn merges: repeatedly glue the most frequent pair of neighbouring tokens into a new token."""
words = Counter(tuple(w.encode()) for w in SPLIT.findall(text)) # each word as a tuple of byte ids
vocab = {i: bytes([i]) for i in range(256)} # start: one token per byte
merges = {}
for new_id in range(256, 256 + num_merges):
pairs = Counter()
for word, freq in words.items():
for pair in zip(word, word[1:]):
pairs[pair] += freq
if not pairs:
break
best = max(pairs, key=pairs.get)
merges[best] = new_id
vocab[new_id] = vocab[best[0]] + vocab[best[1]]
words = Counter({merge_word(w, best, new_id): f for w, f in words.items()})
return merges, vocab
def merge_word(word, pair, new_id):
out, i = [], 0
while i < len(word):
if i + 1 < len(word) and (word[i], word[i + 1]) == pair:
out.append(new_id)
i += 2
else:
out.append(word[i])
i += 1
return tuple(out)
def encode(text: str, merges) -> list[int]:
ids = []
for w in SPLIT.findall(text):
word = tuple(w.encode())
while len(word) > 1:
# apply the earliest-learned merge that is present, like real BPE tokenizers do
pair = min(zip(word, word[1:]), key=lambda p: merges.get(p, float("inf")))
if pair not in merges:
break
word = merge_word(word, pair, merges[pair])
ids.extend(word)
return ids
def decode(ids, vocab) -> str:
return b"".join(vocab[i] for i in ids).decode("utf-8", errors="replace")
A few details:
- Split into words first (
SPLIT). Pairs are only merged inside a word, so there's never a token like"n, t"straddling a comma. Counting by word (aCounterof words) is also much faster than walking the whole text: a word that appears 500 times is looked at once and multiplied by 500.\win Python'sreunderstands Unicode, soệandđcount as letters. - Encoding applies merges in the order they were learned. That's what
min(..., key=merges.get)is for: a merge learned earlier (more common) goes first. decodejust joins bytes. Every token is a byte string, and joining all of them always gives back exactly the original text.
Training on these very posts
The training data is the prose (code removed) of three Vietnamese posts on this site, 37,605 bytes in all. The fourth, One piece at a time, is kept aside as test data the tokenizer has never seen. 2,000 merges take 0.9 seconds. The first twenty tokens learned:
256 '\xe1\xbb' 257 '\xe1\xba' 258 ' t' 259 ' c' 260 'ng'
261 'à' 262 'đ' 263 ' đ' 264 ' l' 265 ' n'
266 ' m' 267 ' th' 268 ' ch' 269 ' v' 270 'ộ'
271 '\n\n' 272 'ư' 273 ' b' 274 ' k' 275 'i\xe1\xbb'
The first two merges are exactly e1 bb and e1 ba, the two byte pairs that begin letters like ạ, ẽ, ệ, ữ. They aren't any complete letter (just two thirds of one), but they're the most frequent byte pairs in Vietnamese text, so BPE merges them first. Next come the initial consonants t, c, th, ch (with the space before them), the ending ng, then the letters à, đ, ộ, ư. Nobody taught the algorithm Vietnamese spelling; it only counts.
Testing on the unseen post (22,405 bytes):
| Merges | Vocabulary | Tokens (Vietnamese post) | Bytes per token | Tokens (English version of the same post) |
|---|---|---|---|---|
| 0 | 256 | 22,405 | 1.00 | 20,153 |
| 100 | 356 | 12,575 | 1.78 | 16,057 |
| 500 | 756 | 8,174 | 2.74 | 13,066 |
| 1,000 | 1,256 | 6,884 | 3.25 | 11,668 |
| 2,000 | 2,256 | 6,251 | 3.58 | 11,191 |
Two takeaways:
- Big gains first, then smaller ones. The first 100 merges cut almost half the tokens; the last 1,000 cut only another 9%. Real tokenizers have hundreds of thousands of tokens to squeeze out this small remaining gain over an enormous amount of text.
- A tokenizer is good at exactly what it learned. Trained on Vietnamese, it needs 6,251 tokens for the Vietnamese post but 11,191 for the English version of the same post, even though the English version is shorter in bytes. Keep this in mind: we'll soon see the same effect, in reverse, in GPT-4's tokenizer.
With 2,256 tokens, the sentence from the start of the post is cut like this:
Tất nhiên, tôi sẽ giúp bạn → 8 tokens: 'T' 'ất' ' nhiên' ',' ' tôi' ' sẽ' ' giúp' ' bạn'
Token for token, identical to how Qwen2.5's 151,665-entry tokenizer cuts this sentence. Learned from 37 KB of text, in under a second.
What real tokenizers add
The tokenizers of GPT-4o or Qwen are still BPE as above, but with a few layers around it. Opening them up:
import tiktoken
from tokenizers import Tokenizer
qwen = Tokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
deepseek = Tokenizer.from_pretrained("deepseek-ai/DeepSeek-V3")
print(qwen.normalizer, "|", deepseek.normalizer)
print(tiktoken.get_encoding("o200k_base")._pat_str)
Normalization
NFC() | Sequence(normalizers=[])
Qwen runs NFC over all text before cutting it, so the decomposed and precomposed forms give the same tokens. DeepSeek-V3 (and OpenAI's tiktoken) normalize nothing: text is cut exactly as it comes in.
Pre-splitting with a regex
The SPLIT pattern above is a simple version of this step. GPT-4o's is much longer:
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|...|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+
Piece by piece: a word may have one non-letter character before it (usually a space), English endings like 's and 'll stay with the word, \p{M} (combining marks) counts as part of a word, and numbers are cut into groups of at most 3 digits (\p{N}{1,3}). Qwen chose differently: \p{N}, one token per digit. These two choices give very different results for numbers; see the consequences section.
Byte-level, and the strange Ġ
Look at Qwen's tokens in raw form (qwen.encode("Xin chào Việt Nam").tokens) and you see:
['X', 'in', 'ĠchÃło', 'ĠViá»ĩt', 'ĠNam']
This isn't a font problem. Hugging Face's library stores tokens as strings, so it maps every byte to a printable character: byte 0x20 (space) becomes Ġ, and the UTF-8 bytes of à and ệ become Ãł and á»ĩ. ĠViá»ĩt is just another way to write the 6 bytes of Việt. Decoding with decode gives normal text back.
Special tokens
The <|im_start|> and <|im_end|> from the start of the post aren't learned by BPE; they're added straight into the vocabulary with their own ids (151644 and 151645). The model learns what they mean during training, and <|im_end|> is also the "end of answer" token: when the model produces it, generate() stops.
They have a dangerous side. What if a user types the exact string <|im_start|> into the chat box?
tokenizer("<|im_start|>", add_special_tokens=False)["input_ids"] # [151644]
tokenizer("<|im_start|>", add_special_tokens=False, split_special_tokens=True)["input_ids"] # [27, 91, 318, 4906, 91, 29]
By default, transformers turns that string into the real special token, so a user can open a fake "system" or "assistant" turn in the middle of their own message. With split_special_tokens=True it's just 6 ordinary tokens <, |, im, _start, |, >. tiktoken takes the safer route: encode("<|endoftext|>") raises ValueError unless you say explicitly how to handle it. If you build prompts from text users type, make sure that text can't become a special token.
How many tokens Vietnamese costs
Now the practical question. Every post on this site has a Vietnamese version and a close English translation, so together they make a fairly fair dataset: same content, two languages. Counting the tokens of the prose (code removed) of all four posts with four tokenizers:
| Tokenizer (used by) | Vocabulary | Vietnamese | English | Vietnamese / English | Characters per token (Vietnamese) |
|---|---|---|---|---|---|
cl100k_base (GPT-4, GPT-3.5) |
100,277 | 21,040 | 12,429 | 1.69 | 2.28 |
o200k_base (GPT-4o) |
200,019 | 14,109 | 12,337 | 1.14 | 3.40 |
| Qwen2.5 | 151,665 | 14,165 | 12,675 | 1.12 | 3.39 |
| DeepSeek-V3 | 128,815 | 18,547 | 12,485 | 1.49 | 2.59 |
For English, all four are nearly even, about 4.1–4.2 characters per token. For Vietnamese, the gap is large: for the same content, GPT-4's tokenizer needs 69% more tokens than for English, GPT-4o's only 14% more. From GPT-4 to GPT-4o, the same Vietnamese posts go from 21,040 to 14,109 tokens, a third fewer.
The reason is the lesson from the BPE section: a tokenizer is good at exactly what it learned. OpenAI hasn't published the data cl100k_base was trained on, but the way it cuts text shows it contained very little Vietnamese. Here's how it cuts a few sentences:
| Sentence | cl100k_base |
o200k_base |
Qwen2.5 |
|---|---|---|---|
| Tất nhiên, tôi sẽ giúp bạn | 16 | 8 | 8 |
| Trường Đại học Bách khoa Hà Nội | 15 | 9 | 10 |
| Tiếng Việt | 6 | 3 | 4 |
| Nguyễn | 5 | 2 | 3 |
| Hello world | 2 | 2 | 2 |
With cl100k_base, "Tất nhiên, tôi sẽ giúp bạn" becomes T ất n hi ên , t ô i s \xe1\xba \xbd gi ú p bạn: the word "tôi" is cut in three, and the ẽ in "sẽ" is split between its bytes. The tokenizer has no token for ẽ, so it falls back to bytes: e1 ba (one of the first two pairs our BPE learned) plus bd. With o200k_base and Qwen, each Vietnamese word is usually exactly one token.
Decomposed and without diacritics
The same Vietnamese posts in precomposed form (NFC, the normal form), decomposed form (NFD), and with every diacritic removed:
| Tokenizer | NFC | NFD | No diacritics |
|---|---|---|---|
cl100k_base |
21,040 | 36,823 | 15,873 |
o200k_base |
14,109 | 30,212 | 14,640 |
| Qwen2.5 | 14,165 | 14,165 | 16,000 |
| DeepSeek-V3 | 18,547 | 32,193 | 15,844 |
- NFD costs double with every tokenizer that doesn't normalize. The vocabulary was learned on precomposed text, so faced with
e+ dot below + circumflex as separate characters, it has no token to combine them:o200k_basecuts "Tiếng Việt" in NFD into 8 tokens (Tie, circumflex, acute,ng,Vie, dot below, circumflex,t) instead of 3. Qwen is immune thanks to its NFC step. If you send Vietnamese text from unknown sources to an API, one line ofunicodedata.normalize("NFC", text)can halve the bill. - Removing diacritics saves nothing with newer tokenizers. With
cl100k_baseit cuts 25%, but witho200k_baseit costs 4% more, and with Qwen 13% more: these tokenizers learned Vietnamese with diacritics as the common form. And that's before "ban" could be "bạn" (you), "bán" (sell), "bàn" (table) or "bận" (busy), leaving the model to guess.
Consequences
Price and context length
APIs charge per token, and context limits ("128K", "1M") are counted in tokens too. The same contract or document in Vietnamese costs 1.69 times the English version on a model that uses cl100k_base: 69% more expensive, and it takes 69% more of the context window. With GPT-4o or Qwen the gap is only 12–14%. When choosing a model for a Vietnamese product, count tokens on your own real text; don't trust a generic "tokens" figure from a price list.
A quick estimate for Vietnamese with a modern tokenizer: about 3.4 characters, or 0.7 words, per token (1.39 tokens per word). For English, the familiar figure is about 4 characters per token.
Speed
A model generates one token at a time, so how much text "50 tokens per second" means depends on the tokenizer. At 3.4 characters per token, that's about 170 Vietnamese characters per second; at 2.28 per token as with cl100k_base, only 114. Same model, same hardware, and a Vietnamese user sees text appear a third slower.
Numbers
| String | cl100k_base / o200k_base |
Qwen2.5 |
|---|---|---|
12345678 |
3: 123 456 78 |
8: one token per digit |
2026-10-17 |
6: 202 6 - 10 - 17 |
10 |
With groups of three, 12345678 is "seen" as 123|456|78, cut from the left, which doesn't match the thousands grouping (12|345|678) that addition needs. The year 2026 becomes 202 + 6. Models still learn arithmetic, but they have to learn it through this odd cutting. Qwen cuts one digit at a time: longer sequences, but the units digit is always its own token, like writing a sum out by hand.
Counting letters
The famous question "how many r's are in strawberry?" used to trip up many models. The reason is right here: in all three tokenizers, strawberry is a single token. The model never sees the letters s-t-r-a-w…; it sees one number, and only knows how to spell it if it learned that from data. Counting letters, reversing a string, finding words that rhyme: things that are easy for people are hard for a model, because the unit it sees isn't a letter.
Streaming: tokens don't line up with characters
Back to the chatbot from the previous post. The model produces one token at a time, and we want to send text right away. The naive way is to decode each token on its own:
import codecs
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode("Nguyễn Văn A sẽ đến")
print([enc.decode([i]) for i in ids])
decoder = codecs.getincrementaldecoder("utf-8")()
print([decoder.decode(enc.decode_single_token_bytes(i)) for i in ids])
['Ng', 'uy', '�', '�', 'n', ' V', 'ă', 'n', ' A', ' s', '�', '�', ' đ', 'ến']
['Ng', 'uy', '', 'ễ', 'n', ' V', 'ă', 'n', ' A', ' s', '', 'ẽ', ' đ', 'ến']
Decoded one token at a time, ễ and ẽ (each split across two tokens) become two � characters each. The user would see "Nguy��n". The incremental decoder (getincrementaldecoder) keeps the unfinished bytes until there's a whole character: the third token gives an empty string, the fourth a complete ễ. It's the same "keep the unfinished piece, wait for more" idea as the buffers in the previous post's streams, just at the byte level.
transformers' TextStreamer (the class the previous post's chatbot inherits from) solves it more crudely. The 5.20 source does this: on every new token, it decodes the whole run again since the start of the line, then only pushes out text up to the last space:
# transformers/generation/streamers.py, TextStreamer.put (abridged)
text = self.tokenizer.decode(self.token_cache)
if text.endswith("\n"):
printable_text = text[self.print_len :] # a full line: flush everything
...
else:
printable_text = text[self.print_len : text.rfind(" ") + 1] # up to the last space only
That's exactly why the previous post's chatbot sent 'Tất ', 'nhiên, ', 'tôi ': each piece is a whole word plus its space, not a token. This avoids broken characters (an unfinished letter never sits before a space), at the price of text arriving one step late: a word is only sent once the next word starts.
Measurements
How fast is a tokenizer? Encoding about 1 MB of Vietnamese text (the posts repeated 17 times), best of several runs:
| Tokenizer | Time for 1.02 MB | Speed |
|---|---|---|
| The BPE above (pure Python, 2,256 tokens) | 521 ms | 2.0 MB/s |
HF tokenizers, Qwen2.5 (Rust) |
175 ms | 5.8 MB/s |
tiktoken o200k_base (Rust) |
22 ms | 46.5 MB/s |
The home-made version is about 24 times slower than tiktoken, but 2 MB/s is plenty for experiments; the tokenizer is rarely the bottleneck. For comparison, the previous post's chatbot generated about 17 tokens per second on a CPU; even the pure-Python version encodes tens of thousands of times faster than that. HF tokenizers is slower than tiktoken partly because it returns more: besides the ids, every encode also computes each token's position (offset) in the original string.
How long does training take? 2,000 merges on 37,605 bytes take 0.9 seconds. The algorithm above recounts every pair after each merge, so it's no use for gigabytes of text and 200,000 merges like a real tokenizer; real libraries only update the pairs affected by the last merge. But the end result is the same kind of table: an ordered list of merges.
Summary
| Concept | What it really is |
|---|---|
| Token | An integer, an index into the vocabulary; the model only ever sees these |
| Chat template | Messages joined into one continuous sequence, marked with special tokens like <|im_start|> |
| Vietnamese in UTF-8 | 1–3 bytes per letter; letters like ạ, ẽ, ệ all start with e1 ba/e1 bb |
| NFC and NFD | Two ways to write the same letter; NFD costs double with tokenizers that don't normalize |
| BPE | Start from bytes, keep merging the most frequent pair; the result is a list of merges |
| A tokenizer is good at what it learned | cl100k_base: Vietnamese costs 1.69× English; o200k_base: 1.14× |
| Numbers | Groups of 3 digits (OpenAI) or one digit at a time (Qwen) |
| Counting letters | strawberry is one token: the model doesn't see letters |
| Streaming | A token can end in the middle of a character; decode incrementally, or wait for a space like TextStreamer |
At the end of this pipeline is a sequence of numbers. The next post follows it into the model: each id is looked up as a vector, the vectors "look at" each other through attention, and out comes a probability distribution over the next token. We'll write a small GPT to see what one step of generating one token actually computes.