The previous five posts covered the whole path inside: text into tokens, tokens through the model, why it's fast or slow, where the weights come from, and how many people are served. This last post turns to the people building products, with one observation underneath everything: the model only knows how to do one thing, which is guess the next token. Everything else an "AI application" can do (follow instructions, calculate exactly, look up data, work in several steps) is text placed around the model, and code running around that text.
The route: what a system instruction really is, letting the model call tools, the agent loop (and where a small model gets it wrong), RAG so the model answers from your data, what each technique costs, then a look back at the whole series from a product's point of view.
Every number in this post was measured on a real machine, with Python 3.12, PyTorch 2.14, transformers 5.20, and the Qwen2.5-0.5B-Instruct model on the CPU with 2 threads. Every answer is greedy generation with the repetition penalty switched off (the trap from part 2), so running it again gives exactly the same output.
A system instruction is just text
Part 1 showed that when you send "Xin chào" ("Hello"), what goes into Qwen is 32 tokens, including a default instruction: You are Qwen, created by Alibaba Cloud. You are a helpful assistant. That's the system prompt. It has no special power: it's the tokens at the start of the sequence, inside a <|im_start|>system frame. The model follows it because during fine-tuning (part 4), it saw many conversations where the answer obeyed the system part.
Two practical consequences:
- Instructions cost tokens on every request. A 1,000-token system prompt is 1,000 tokens of prefill (part 3) for every question. Put the fixed part first to benefit from prompt caching.
- Instructions aren't a fence. Users can write anything in their messages, including "ignore all previous instructions". The model only sees a sequence of tokens, and nothing guarantees the system part always wins. Every real limit (who may see what data, who may do what) has to live in your code, not in the prompt.
Tools: let code do what the model does badly
Ask Qwen a multiplication:
Q: 1234 nhân 5678 bằng bao nhiêu? (What is 1234 times 5678?)
A: ... 1234 x 5678 = 6,888,032 ...
The right answer is 7,006,652. Part 1 explained part of the reason: Qwen cuts numbers into single digits, and the model has to "do it in its head" by guessing each token of the result, with no multiplication table inside. A wrong answer, given with great confidence.
The fix isn't a model that's better at arithmetic; it's letting code do the arithmetic. We describe a function to the model:
tools = [{"type": "function", "function": {
"name": "multiply", "description": "Multiply two numbers exactly.",
"parameters": {"type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}, "required": ["a", "b"]},
}}]
inputs = tokenizer.apply_chat_template(messages, tools=tools, add_generation_prompt=True, return_tensors="pt", return_dict=True)
How does that description get into the model? Look at the text the chat template produces:
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "multiply", "description": "Multiply two numbers exactly.", ...}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
1234 nhân 5678 bằng bao nhiêu?<|im_end|>
<|im_start|>assistant
A "tool" is just text in the system prompt. A paragraph of instructions in English, the function described as JSON, and a formatting convention. The prompt goes from 44 tokens to 187. The model replies:
<tool_call>
{"name": "multiply", "arguments": {"a": 1234, "b": 5678}}
</tool_call>
The model doesn't run anything. It only produces tokens that look like a function call, because it learned this convention during fine-tuning. Running the function is our code's job: find the <tool_call> block, read the JSON, call the real function, then hand the result back to the model. APIs such as OpenAI's or Anthropic's return tool calls as structured data rather than raw text, but underneath, the model is doing exactly this: producing tokens in a learned format.
The agent loop
Putting it together: ask the model; if it asks for a tool, run the tool, add the result to the conversation, and ask again; repeat until it answers without asking for anything.
import json
import re
TOOLS = {"multiply": lambda a, b: a * b, "add": lambda a, b: a + b}
CALL = re.compile(r"<tool_call>\s*(\{.*?\})\s*</tool_call>", re.S)
def agent(question: str, max_turns: int = 5) -> str:
messages = [{"role": "user", "content": question}]
for _ in range(max_turns):
reply = complete(messages) # apply_chat_template(..., tools=SCHEMAS) + generate, as above
calls = [json.loads(c) for c in CALL.findall(reply)]
messages.append({"role": "assistant", "content": reply})
if not calls:
return reply # no tool requested: this is the answer
for call in calls: # the model asked; *our* code runs the tool
result = TOOLS[call["name"]](**call["arguments"])
messages.append({"role": "tool", "content": json.dumps(result)})
return reply
The tool's result goes back to the model as text too. With Qwen, it becomes a new user turn:
<|im_start|>user
<tool_response>
7006652
</tool_response><|im_end|>
<|im_start|>assistant
Running three questions, with two tools, multiply and add:
| Question | The model calls | Answer |
|---|---|---|
| 1234 nhân 5678 bằng bao nhiêu? (1234 × 5678?) | multiply(1234, 5678) → 7006652 |
"… là 7006652." ✓ |
| Thủ đô của Việt Nam là gì? (Capital of Vietnam?) | (nothing) | "Thủ đô của Việt Nam là Hà Nội." ✓ |
| Tính (12 + 30) nhân 7. (Compute (12 + 30) × 7.) | multiply(12, 30) → 360, then add(7, 0) → 7 |
"The result … is 360." ✗ |
The first two are right: the model knows when it needs a tool and when it doesn't. The third shows the limit: the calculation needs two steps in the right order (add first, then multiply), and a half-billion-parameter model can't make that plan. It multiplies the first two numbers by mistake, adds two meaningless numbers, then answers in English with a wrong result (the right one is 294). The format is perfect; the reasoning isn't. Bigger models are far better at breaking a problem into steps, and that's exactly what makes "agents" (a model deciding its own sequence of actions) usable in practice.
A few things every real agent loop needs:
- A turn limit (
max_turns). A model can keep calling tools forever. - Checks before running. The JSON the model produces may be malformed, call a function that doesn't exist, or pass dangerous arguments. The model only suggests; your code decides whether to run it, and with what permissions. A "delete file" or "send email" tool deserves the same treatment as an input field filled in by a stranger.
- Every tool result costs tokens. Each turn, the whole conversation (including earlier calls and results) goes through prefill again.
RAG: answering from your data
Ask Qwen about this very website:
Q: Website này chạy sau reverse proxy nào? (Which reverse proxy does this website run behind?)
A: Website này chạy sau reverse proxy là Nginx. Nginx là một trong những công cụ reverse proxy
phổ biến nhất ... (It runs behind Nginx. Nginx is one of the most popular reverse proxies ...)
Q: Huấn luyện mô hình nhỏ trên Wikipedia tiếng Việt mất bao lâu? (How long does training the small model on Vietnamese Wikipedia take?)
A: Huấn luyện mô hình nhỏ trên Wikipedia tiếng Việt mất khoảng 1-2 ngày. ... (About 1–2 days. ...)
Both wrong, and wrong with great confidence. The model has never read these posts, so it guesses from what's most common (Nginx is a popular reverse proxy; "training a model" sounds like it takes days). It's the same mechanism as "Johny von Neumann" in part 2: the model picks plausible tokens; it doesn't look up facts.
RAG (retrieval-augmented generation) fixes this in the simplest possible way: before asking, find the relevant pieces of your documents, then paste them into the prompt. The model doesn't need to know the answer; it only needs to read and copy.
Searching with BM25
The "find" part needs no model at all. BM25 is a classic ranking formula from search engines: a passage scores high if it contains many of the question's words, rare words count more than common ones, and long passages are penalized so they don't win just by being long.
import math
import re
from collections import Counter
WORD = re.compile(r"\w+")
class BM25:
"""Score pieces by shared words, weighting rare words more and long pieces less."""
def __init__(self, docs, k1=1.5, b=0.75):
self.docs = docs
self.tf = [Counter(WORD.findall(d.lower())) for _, d in docs]
self.len = [sum(t.values()) for t in self.tf]
self.avg = sum(self.len) / len(self.len)
df = Counter(w for t in self.tf for w in t)
self.idf = {w: math.log(1 + (len(docs) - n + 0.5) / (n + 0.5)) for w, n in df.items()}
self.k1, self.b = k1, b
def search(self, query: str, k: int = 3):
words = WORD.findall(query.lower())
scores = []
for i, tf in enumerate(self.tf):
norm = self.k1 * (1 - self.b + self.b * self.len[i] / self.avg)
scores.append(sum(self.idf.get(w, 0) * tf[w] * (self.k1 + 1) / (tf[w] + norm) for w in words if w in tf))
best = sorted(range(len(scores)), key=scores.__getitem__, reverse=True)[:k]
return [self.docs[i] for i in best]
The documents are the prose (code removed) of every Vietnamese post on this site, cut into pieces of about 800 characters: 189 pieces. Indexing takes 4 ms; each search takes under 1 ms. The three top-scoring pieces go into the system prompt, along with an instruction ("Answer briefly in Vietnamese, using only the documents below. If the documents don't contain the answer, say you don't know."):
context = "\n\n---\n\n".join(text for _, text in index.search(question))
messages = [
{"role": "system", "content": "Trả lời ngắn gọn bằng tiếng Việt, chỉ dựa vào tài liệu dưới đây. "
"Nếu tài liệu không có câu trả lời, hãy nói không biết.\n\n" + context},
{"role": "user", "content": question},
]
| Question | Without RAG | With RAG | Pieces found in |
|---|---|---|---|
| Which reverse proxy does this website run behind? | Nginx ✗ | "Website này chạy sau Caddy." (It runs behind Caddy.) ✓ | the streaming post |
| How long does training the small model on Vietnamese Wikipedia take? | 1–2 days ✗ | "… mất khoảng 1 phút." (about 1 minute) ✓ | the training post |
| With GPT-4's tokenizer, how many times more tokens does Vietnamese cost than English? | "I'd need to know exactly…" | "… tốn gấp 69% so với tiếng Anh." (costs 69% times English) ~ | the tokens post |
The first two go from wrong to right. The third is nearly right: the document says "69% more" (1.69 times), and the small model copies it as a garbled "69% times". That's the thing to remember about RAG: it solves the model doesn't know, but not the model reads poorly. Bigger models read and restate far more accurately; with any model, it's worth showing users the source passage so they can check.
BM25 matches the same words. If a user asks about a "máy chủ trung gian" ("intermediary server") instead of a "reverse proxy", it may miss. RAG systems often add search with embeddings: a model turns each passage into a vector so that passages with the same meaning sit close together, to search by meaning rather than by spelling. The two are often used together, because BM25 is good with names, product codes and numbers, things embeddings tend to blur.
What RAG costs
| Without RAG | With RAG | |
|---|---|---|
| Prompt tokens | 37–55 | 657–898 |
| Prefill (time to first token) | 71–103 ms | 481–682 ms |
Each question now carries about 2,500 characters of documents, so the prompt is 16–18 times longer, and the first token arrives 6–8 times later. This is part 3 applied directly: prefill grows with prompt length, and with an API, input tokens are money. That's why the quality of the "find" step matters as much as the "generate" step: finding the right 3 pieces beats pasting in 30 "just in case".
Putting it in an endpoint
Every piece of this post fits neatly into the chatbot from the Python series. Its answer function takes a question and streams tokens to the browser; only the line that builds inputs changes:
async def answer(question: str):
context = "\n\n---\n\n".join(text for _, text in index.search(question)) # retrieval: under 1 ms
messages = [
{"role": "system", "content": RAG_INSTRUCTIONS + "\n\n" + context},
{"role": "user", "content": question},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
... # the rest is unchanged: model in a thread, bounded queue, async generator, SSE
The rest stays the same: the model runs in a thread, tokens go through a bounded queue, finally stops the model when the user leaves the page. With an agent loop, the tool-calling steps run first and only the final answer is streamed; or you stream the steps too ("calculating 1234 × 5678…") so users can see what the system is doing.
Looking back at the series
Every decision when building a product on an LLM comes back to one of the five earlier posts:
| The product question | Where the answer is |
|---|---|
| How much will it cost? How long is too long for a prompt? | Part 1: tokens; Vietnamese and NFC/NFD |
| Why do answers differ between two places? What temperature? | Part 2: sampling, generation_config.json |
| Why is the first token slow? What hardware should we use? | Part 3: prefill, decode, memory bandwidth |
| Why does the model make things up? What does it know and not know? | Parts 2 and 4: guessing tokens, training data |
| What does serving many people cost? Should we quantize? | Part 5: batching, the KV cache, 8 and 4 bits |
| How do we make it calculate right, use our own data, work in steps? | This post: tools, RAG, agents |
Summary
| Concept | What it really is |
|---|---|
| System prompt | The tokens at the start of the sequence; the model follows it because it learned to, not because it's forced to |
| Tools | A JSON description inserted into the system prompt; the model produces <tool_call>, your code runs the function |
| Tool results | A new turn in the conversation, as text (<tool_response>) |
| Agent | A loop of ask → run tools → ask again; needs a turn limit and a check on every call |
| A small model's limit | The format is right, the multi-step plan is wrong |
| RAG | Find the relevant pieces and paste them into the prompt; fixes "doesn't know", not "reads poorly" |
| BM25 and embeddings | Search by words and search by meaning; often used together |
| The cost | RAG makes the prompt 16–18 times longer and the first token 6–8 times later |
The series began with "the model doesn't read text, it reads numbers" and has now reached an application that can calculate and look things up. In between is a chain of ideas with nothing mysterious about it: a lookup table, a few matrix multiplications, a loop that guesses tokens, and a lot of text put in the right place.
The last post looks at what surrounds today's agents: skills, subagents, MCP and hooks, the things that turn the 20-line loop above into a real working tool.