Chapter 22 What Makes a Language Model an LLM

Our Mini GPT can complete a familiar red-key sentence. It can also answer red after the prompt explicitly changes the key to blue. Both observations come from the same final snapshot in Chapter 21. The successful completion shows that a learned distribution can reproduce a pattern; the changed-clue failure limits what we can say about that pattern. Neither putting the model in a chat window nor calling it intelligent changes this evidence.

Modern large language models share that next-token computation with Mini GPT, but the surrounding conditions can be very different: which texts were available, how much training occurred, what capacity the model had, and what later training encouraged it to do. To compare them, we need several questions rather than one size threshold. A model card is a versioned description of a model’s construction, intended uses, evaluations and limitations. We will read one by connecting its claims to evidence.

22.1 Begin with the model we can actually inspect

Example 22.1 — Reconstruct MG-2 from evidence. The input is the unchanged MG-2 bundle. Compare the validation-selected step-50 snapshot, then the final step-600 snapshot. Both have 13,875 learned scalars, one causal block, a 291-entry byte-BPE vocabulary and 64 input slots including BOS. They share 24 training records and eight validation records. Their weight values differ.

Table 22.1 — Same structure, different evidence. Loss is in nats per valid next-token target; completions preserve the measured spelling. Each displayed prompt ends with a space after its colon.

Evidence	Step 50	Step 600
Validation loss	2.405571	4.068374
The key is red. Answer:	bluesuersther: blue key.	red.
The key was red. Now blue. Answer:	blue key.	red.
Old limit: 600. New limit: 750. Answer:	peng.	shipedwion: old.

The first row favors step 50 under the declared validation rule. The second favors step 600 if the sole task is reproducing that familiar suffix. The other rows warn against treating either as a reliable updated-information assistant. A model card must say which question each evaluation answers. Averaging unrelated rows into an unnamed “intelligence score” would remove the distinction we need.

More parameters alone would not repair the experiment. We could duplicate an unused parameter array, increase the count, and leave every output unchanged. Even parameters used by the forward pass need suitable data and optimization before they support a useful behavior. Scale concerns capacity and the resources used to learn, not a certificate obtained by crossing a single integer.

22.2 Ask separate questions before attaching names

First ask whether one trained model is intended to support many downstream uses. A downstream use is a task built on the trained model, such as summarizing a policy or classifying a request. Foundation model emphasizes broad training and reuse through adaptation. The term can include models for images or other modalities, so it does not mean “text chatbot.” MG-2’s narrow original corpus and diagnostics do not establish that breadth.

Next ask about language modeling at substantial scale: parameter capacity, training-token exposure, data diversity and demonstrated capabilities. Large language model (LLM) is the broad label used for such language models. There is no universal parameter cutoff adopted here. A count helps compare specified architectures; it does not replace the other fields. A smaller member of a modern model family can still be called an LLM, and different naming practices do not settle its suitability for a task.

Now ask where the saved weights sit in the training process. A base model is a pretrained model before the particular assistant-oriented adaptation under discussion. For a causal language model, its ordinary objective is predicting the next token in text. That text includes answers, questions, quoted mistakes and unfinished conversations. The objective does not itself say which speaker should obey a user’s request. An instruct model has received training aimed at following instructions. This can change what behavior training rewards without changing the number of layers or parameters.

Finally ask two different behavioral and delivery questions. Chat may describe a model adapted for multi-turn dialogue, a dialogue template, or the interface of an application. The surrounding documentation must disambiguate it. Reasoning model usually indicates a model trained or configured to improve multi-step problem-solving, often with additional computation before an answer. It is a capability/training emphasis, not a distinct mathematical alternative to running the model. Longer visible explanations alone do not establish correct reasoning, and a generated explanation is not an authenticated record of internal operations.

Table 22.2 — Labels answer different questions. These rows are independent axes, not mutually exclusive boxes.

Question	Relevant label	Evidence still needed
Can a broad pretrained model be adapted across tasks?	Foundation	Data scope and evaluations across intended uses
What is the scale and scope of language modeling?	LLM	Parameter definition, token budget, mixture, capabilities
Which point in training supplied these weights?	Base / instruct	Parent snapshot, objectives and additional training
Was dialogue behavior trained, or is there only a dialogue interface?	Chat	Template, provenance and multi-turn evaluation
Is multi-step problem-solving an explicit emphasis?	Reasoning	Task, budget, verification and failure evidence

A model can therefore be a foundation language model, an LLM, an instruct model and a reasoning-oriented model at the same time. Calling a model “base” does not say it has no useful abilities. Calling another “instruct” does not guarantee that its answers are true. The instruction-training and reasoning papers in the references demonstrate particular lifecycles; they do not define one mandatory recipe for every model.

22.3 Read a card as a chain of claims

Example 22.2 — A vendor-neutral card with an incomplete evaluation. This is an author-assigned exercise, version lifecycle-card-v1. The numbers describe a hypothetical system, not a measured model release. What can we reconstruct, and what remains unknown?

Table 22.3 — The card and its immediate implications. A billion means 1,000,000,000. A training token here is one processed token exposure under the stated tokenizer, including repeated text.

Field	Card entry
Architecture	Causal dense Transformer; 1 billion total parameters; tokenizer T1
Pretraining	Broad mixed text, 20 billion token exposures; mixture documented separately
Base snapshot	B1, after next-token training
Adapted snapshot	I1, initialized from B1, instruction demonstrations and preference training; same parameter count
Evaluation	English policy questions reported; no Chinese evaluation; changed-policy test omitted
Delivery	I1 through engine E1 and server S1; browser chat interface
Reasoning claim	Marketing description only; no budget or verified problem set
Use boundary	Draft recommendations; no order execution; deployment license still to be checked

We can reconstruct the claimed lineage B1 → I1 and locate additional training between them. We cannot infer from “20 billion” that there were 20 billion distinct pieces of information: documents may repeat. Nor can we infer that every source was legally reusable or every language adequately represented. Those questions require the data record.

The English result cannot fill the missing Chinese result. The unchanged parameter count cannot show that B1 and I1 produce the same distribution: training can change stored values while preserving every tensor’s shape. Conversely, replacing the browser interface can leave all those values untouched. The reasoning claim remains unsubstantiated under this card. That is a legitimate reading result, not a blank we should fill with optimism.

The shared data/part-v/lifecycle-card.json fixes this exercise. For a real candidate, also retain a stable revision, tokenizer identity, tested prompt/template, decoding configuration, intended use and applicable license. A mutable name without a revision makes a later comparison difficult to repeat. A card organizes evidence; it does not automatically verify its author’s claims.

22.4 Follow what changes from data to a served answer

In pretraining, a pipeline selects text and turns it into token sequences. The training program computes loss and updates parameters, as MG-2 did. Saving a base snapshot freezes one point in that process. Optional post-training begins from a snapshot and updates parameters using another objective or data distribution. Continued pretraining and repeated rounds are also possible; this is a dependency diagram, not an assertion that every model follows one linear historical path.

Parameter-changing path

1. Outside the modelSelected text → token sequences
2. Stored model numbersNext-token updates → B1
3. Stored model numbersAdditional training → I1

B1 and I1 have the same shapes; their stored values can differ.

Parameter-reading path

1. Stored model numbersLoad I1 + tokenizer T1
2. An operationEngine E1 computes logits
3. Outside the modelServer S1 schedules requests

A new server or window does not establish a new training stage.

One application request

1. Text piecesContext includes new limit: 750
2. An operationRead frozen weights → select tokens
3. Outside the modelDisplay answer; no order is changed

Context may change the output while parameter identity stays fixed.

Figure 22.1 — Training changes parameters; serving reads a snapshot. Original book diagram · CC BY-SA 4.0.

Figure 22.1 separates parameter-changing arrows from parameter-reading operations. Once the chosen snapshot is loaded, an engine performs its numerical operations and token selection. A server accepts requests and allocates resources. An application assembles input, presents results and decides whether some separate program may act. Chapters 30–31 examine the first two responsibilities. Later parts develop application interfaces and permissions.

Suppose a chat page adds “Please answer politely” to the input while retaining B1. The model sees a different context, which can change its output, but no post-training has occurred. Suppose instead that the page looks identical while its backend loads I1. The interface stays fixed while the parameter distribution changes. In both situations an answer about cancellation is still text; the model has not obtained authority to cancel order A-104.

This also explains why a training-data cutoff is not the date of every answer. A fixed model can receive the new fictional 750-yuan policy in its current input. Whether it uses that evidence correctly is a behavioral question. It need not have been retrained to see it, and seeing it does not guarantee that it follows it. MG-2’s explicit-value failure supplies a small counterexample.

22.5 Summary

Mini GPT gives us an inspectable training and generation mechanism, together with evidence that its learned patterns are narrow. Modern LLMs add different scales, data conditions and training processes, not an automatic promise of truth or authority. Foundation, LLM, base, instruct, chat and reasoning describe separate aspects of a candidate. Connect each label to its lineage, inputs, evaluation and missing evidence. The next chapter follows the data before it becomes a training token.

22.6 Exercises

1. A team adds a chat window and a polite system prompt to unchanged MG-2 step-600 weights. Which objects changed? Does this establish instruction training?
2. Two cards report one billion parameters. One counts only active experts, the other all stored parameters. Can the counts be compared directly? Identify the missing definition without needing the expert implementation yet.
3. The I1 card adds a Chinese model name but no new evaluation. A buyer concludes that Chinese and English quality are comparable. What evidence is missing, and what small test could reveal a difference?
4. A model answers one arithmetic question correctly after producing a long explanation. Give one supported observation and two claims this cannot establish.
5. B1 was trained before travel-v2 became effective. An application supplies the new policy at request time. Must a correct 750 answer imply updated parameters? What information would distinguish the mechanisms?

22.7 Reference answers

1. The interface and input-construction rule changed; the stored weights did not. The prompt can affect a run, but the team has shown no instruction-oriented parameter update. Record the prompt and wrapper version separately from the model revision.
2. No. “Active” counts the portion used on a particular token path; “total” counts the stored model. Ask for both, their treatment of shared/embedding parameters and the routing assumption. Chapter 26 calculates an example. Equal printed numbers with different definitions are not equal resources.
3. A name supplies no language-stratified evidence. Use corresponding Chinese and English updated-limit questions, the same information and acceptance criteria, and record failures and tokenizer lengths. This diagnostic can reveal a difference; it cannot establish representative bilingual quality by itself.
4. The observed run produced a correct answer to that question within its recorded budget. It does not establish reliable general reasoning or prove that the visible explanation faithfully reports the internal causal process. Changed problems and controlled evaluations are needed for broader claims.
5. No. The unchanged model may use supplied context. Compare weight/tokenizer identities and exact inputs, including any retrieved documents. Removing or changing the supplied policy in a controlled comparison helps investigate dependence on context, while still requiring careful interpretation.

22.8 References

• Bommasani et al., On the Opportunities and Risks of Foundation Models. Sections 1 and 4.2 explain the broad-training/adaptation perspective; they do not specify a universal LLM parameter cutoff.
• Mitchell et al., Model Cards for Model Reporting. Section 4 supplies reporting dimensions for interrogating the original exercise card.
• Ouyang et al., Training language models to follow instructions with human feedback. Figure 2 and the evaluation discussion distinguish additional behavior-oriented training from the original objective, within a specified prompt distribution.
• DeepSeek-AI, DeepSeek-R1, v1. The overview supplies one reasoning-oriented lifecycle; its recipe and benchmark claims are not a universal definition.