Chapter 1 Start by Observing: What an LLM Is—and Is Not

1.1 The problem: what would count as a correct answer?

When you type a question into a chat window, the software returns a reply. We will call that complete service the assistant. The words sent to it are an input; the words it returns are an output, or response. Inside the service, a model performs the numerical computation used to generate text. The surrounding application handles such tasks as displaying the conversation and retrieving files. We will examine their division of work through examples; no programming knowledge is needed.

Consider an assistant used to prepare a business trip. It completes a sentence about France, reports a lodging allowance, and checks an order. These requests share a conversational interface, but they require different kinds of evidence. Geography can be checked against a factual reference. A reimbursement answer depends on an organization’s applicable policy. An order answer depends on a particular order record. A convincing writing style cannot supply all three kinds of evidence.

Our starting problem is therefore more precise than “Can the assistant answer questions?” It is: given a task, an input, and a response, how can we establish whether the system did what was required? We must answer this before attempting to explain how the model works. The first chapter asks how to judge a reply; Chapter 2 follows how one is produced. Chapters 3 and 4 then let us carry out two pieces of that process ourselves: turn text into numbers, and use counts to predict what comes next.

A task specifies a requested result and any constraints on obtaining it. An evaluation criterion specifies how we judge the result. For “name the capital of France,” the relevant criterion is factual agreement. For “write a plausible sentence ending,” several different outputs may satisfy the criterion. For “check an order,” the method matters too: the system must consult the right record and must not alter the order merely to report its status.

This chapter uses five pairs of fixed, authored teaching records, covering factual completion, open-ended completion, clue substitution, policy updates, and order queries. They are not outputs measured from a real model. The policy, person, and order are fictional; the geography fact is ordinary factual background. The records deliberately isolate different questions about correctness. Their complete conditions and replies are given below, so the analysis requires neither an account nor a running model. Each record describes one input, the conditions under which it is answered, and one reply. A pair lets us compare what happens when a relevant condition or reply changes. Readers inspecting the accompanying data will find the same records under version part-i-v1.

The distinction between a constructed example and a measurement will matter throughout the book. An example can establish what follows from its assumptions. To estimate a real system’s reliability, we would need observations from that system under a specified sampling and evaluation procedure.

1.2 Establish a criterion before comparing responses

1.2.1 Factual completion: grammatical fit is insufficient

Example 1.1 — Two continuations of the same input. Both records receive exactly The capital of France is ___. No tool is available. The responses are:

Record	Response	Factual criterion	Result
First	Paris.	Name the capital: Paris	Pass
Second	Lyon.	Name the capital: Paris	Fail

Both city names fit the sentence’s grammar. The error in the second response therefore cannot be detected by checking only sentence structure. We must compare the asserted relationship—France has this capital—with an independent reference.

This gives us two different properties: a continuation can be linguistically plausible and factually incorrect. A generated statement presented as a fact without adequate support is often discussed under the broad label hallucination. That label alone does not tell us which claim failed or how to repair it. “The answer names Lyon as the capital of France” is a more useful failure description because it identifies a checkable proposition.

Language modeling learns regularities in text, including many useful factual associations. The problem is that text also contains mistakes, fiction, outdated information, and common misconceptions. Predicting a familiar continuation does not automatically distinguish these categories. TruthfulQA investigated false answers that resemble human misconceptions; its results motivate treating truth as a separate evaluation question. Our Paris/Lyon pair is an original illustration, not an example taken from that benchmark. Lin et al., 2022

The correct response, Paris., establishes only that this answer agrees with the criterion for this case. It does not establish that the same system can locate a company’s current policy, use a distant clue, or carry out an authorized operation. These are additional tasks with additional conditions.

1.2.2 Open-ended completion: exact agreement can be too restrictive

Now consider the input Finish: The cat sat on the ___. We have not identified a real event or required a particular noun. Both mat and sofa make plausible completions. If an evaluator stores only mat as its accepted string, the evaluator will reject a valid response.

Having two acceptable endings raises another question: how can the assistant choose between them? One rule would always choose the same ending. Another would allow both, while making one more likely. To picture the second rule, put ten equally likely tickets in a bag: six say mat and four say sofa. Draw one ticket without looking. After each draw, put it back and mix the tickets again, so the chances stay the same on the next draw. The chance of mat is 6 out of 10, written 0.6 or 60%; the chance of sofa is 0.4 or 40%. These probabilities describe the selection rule. They do not promise exactly six mat results in the next ten draws, and they do not measure whether either sentence is true.

Example 1.2 — Variation under a fixed distribution. Assign probability 0.6 to mat and 0.4 to sofa. This complete list of choices and their probabilities is a probability distribution. Instead of physical tickets, use a number line of length 1. Give the first 0.6 of its length to mat and the remaining 0.4 to sofa. Choose a number uniformly from 0 inclusive to 1 exclusive: intervals of equal length are equally likely to contain it. A number below 0.6 selects mat; a number at least 0.6 selects sofa. This implements the same chances as the tickets.

Table 1.1 — Two draws from the same distribution.

Record	Available choices and probabilities	Draw	Selected response	Task result
First	mat: 0.6; sofa: 0.4	0.2	mat.	Pass
Second	mat: 0.6; sofa: 0.4	0.8	sofa.	Pass

Selecting according to a probability distribution is sampling. In this construction, changing the draw is sufficient to change the response; the distribution stays fixed. It follows that two different responses do not, by themselves, establish that the model changed.

The criterion is equally important. Under the different instruction “Repeat the word mat exactly,” sofa would fail. Under that instruction, even adding a full stop could violate a strictly specified character-for-character requirement. There is no universal rule that variation is good or bad. Acceptability depends on what the task permits.

In a real experiment, changed instructions, documents, model versions, or selection settings can also change the response. To attribute variation to sampling, those other conditions must be controlled or recorded. Here the cause is known because the distribution and both draws were specified when constructing the example.

1.3 Test whether an answer depends on the relevant input

The first two cases asked whether a response was acceptable. We now ask whether it follows the information in the question. Suppose an assistant always answers “red” when asked about a key. It will pass a story about a red key even if it ignores the story entirely. To tell that behavior apart from using the clue, we need a second story whose answer is different.

Example 1.3 — A controlled clue substitution. The first record receives:

Mira put a red key in the drawer. The lamp was off. A train passed. Someone closed the window. What color was the key?

The second record receives the same text with only red replaced by blue:

Mira put a blue key in the drawer. The lamp was off. A train passed. Someone closed the window. What color was the key?

Both stored responses are Red. The first agrees with its input; the second contradicts its input. The lamp, train, and window sentences are unchanged, so they do not explain a change in the required answer.

The supplied story and question are part of the context: information made available for the current model run. In the clue-substitution case, the answer should depend on one detail in that context. The substitution creates a contrastive case: the two inputs are almost identical, but the correct answers differ for a known reason. Simply repeating the earlier answer can no longer pass both cases.

This test tells us what went wrong in the response: it did not correctly use the supplied blue clue. It does not yet tell us why. The application might have omitted the clue when preparing the input; the model might have received it and still answered incorrectly. The same visible error can arise at different steps, so we must inspect those steps before choosing a remedy.

A follow-up investigation should distinguish at least two possibilities. First, did the application actually send the blue clue to the model? A displayed conversation may differ from the selected input. Second, if the clue was sent, does its position affect the result? One could keep the blue clue and all other sentences fixed while moving the clue immediately before the question. Inspecting the actual input addresses the first question; a controlled position comparison addresses the second.

Research on long contexts has found performance sensitive to where relevant information occurs. That supports investigating placement, but our short invented story measures no real model’s long-context ability. Liu et al., 2023

There is a general experimental principle here. A useful counterexample does more than produce another failure: it distinguishes explanations that the original case could not separate. The red story alone cannot distinguish successful clue use from a habit of answering red. The blue story can.

1.4 Locate the evidence outside the model

1.4.1 A current answer requires a current source

The key-color story provides the needed fact in the input. A policy question presents a different problem: the relevant information may exist elsewhere and may have changed.

A familiar comparison is an open-book question: a newly revised page helps only if the person answering actually receives and uses it. A file stored somewhere on a computer is likewise not automatically part of the text sent to the model.

Our fictional organization has two policy versions. travel-v1 sets the Beijing lodging limit at 600 CNY per night. travel-v2 replaces v1 and sets the limit at 750. Here v1 and v2 simply mean “version 1” and “version 2”; the labels let us track which document was used. The task is to state the applicable current limit, not to recall any number that once appeared in a policy.

Example 1.4 — Source availability and source selection. The following records keep the current external document at v2, but differ in what enters the input.

Table 1.2 — Policy answers with and without the current excerpt.

Record	Exact input	Source supplied to the run	Response
First	What is the nightly Beijing lodging limit?	None; v2 exists externally	600 CNY per night.
Second	travel-v2: Beijing lodging limit is 750 CNY per night, replacing travel-v1 (600). What is the nightly Beijing lodging limit?	travel-v2	750 CNY per night, according to travel-v2.

The first response fails the current-policy task. The second agrees with the supplied current excerpt and identifies its version. The improvement in this constructed pair comes with a concrete change: the new rule has been made available in the selected input.

Three propositions must be kept separate: a current document exists; the application selected that document; the response correctly used its contents. None of the first two propositions alone establishes the third. A system could select the right document and misread it. It could also produce 750 by chance while failing to identify any valid basis for that number.

If no current evidence is available to the run, a response such as “I cannot verify the current limit from the supplied information” satisfies the evidence boundary better than an unsupported current-policy claim. It leaves the user’s task unresolved, but accurately identifies what is missing. This separates task completion from responsible handling of insufficient evidence; an evaluation should record both rather than pretending that abstention and a verified answer are identical outcomes.

Retrieval-augmented systems use surrounding software to find external material and supply it for generation. The RAG paper develops an architecture that combines learned model information with retrieved documents. Supplying our short excerpt illustrates the separation, while leaving retrieval and document processing for later chapters. Lewis et al., 2020

1.4.2 A claim about an action requires evidence of execution

The policy case needed a document. Checking an order requires a different source: the program that maintains order records. This introduces a second distinction. Reading an order tells us its status; changing the order can affect what happens to the purchase. A reply in the chat window alone does neither.

For an order, the source is a record maintained by another program. The relevant record in our example is order-v1: order A-104 has status processing.

Example 1.5 — Reporting a state is different from changing it. The first record receives Check the status of order A-104. It has no tool connection and no lookup result, yet returns A-104 has shipped. The second record receives the same request followed by Application supplies order-v1: order_id=A-104, status=processing. It returns A-104 is processing, according to order-v1.

Both records specify zero order mutations. The first response still fails: preserving the order does not make its unsupported status claim correct. The second response meets both requirements of this case: report the supplied status accurately and leave the order unchanged.

Now change the request to “Cancel A-104.” This asks for a state change. Our application requires approval before cancellation. A successful result would require an authorized request, the required approval, an executed cancellation operation, and evidence that it succeeded. Merely generating the sentence “A-104 is cancelled” supplies none of the missing execution evidence.

This distinction also applies when a model produces a structured tool request. A tool is an operation exposed by surrounding software, such as reading an order. The model can propose arguments for that operation; application code must decide whether to execute it and then return the result. ReAct makes the separation between proposed actions and environmental observations explicit. Our approval requirement is an application rule, not a property guaranteed by that research method. Yao et al., 2023

An unsupported claim and a false claim are related but different. If the order had happened to ship, the first response could have been factually correct while still lacking a demonstrated lookup. For a task that requires verified status, that missing evidence would remain a defect. Evaluation must check the required procedure as well as the final wording.

1.5 Distinguish the model from the system

Return to the sentence about the cat. We gave mat and sofa probabilities by hand. A language model provides probabilities through its own computation, based on what came before. To generate a longer reply, we can choose a piece, append it to the input, and calculate again. Repeating this step builds text piece by piece.

More generally, a language model assigns probabilities to sequences of language units. The text generators studied in this book commonly do so by predicting the next unit from the preceding input. The units are called tokens; they may be words, parts of words, punctuation, or other pieces determined by an encoding scheme.

A large language model (LLM) is a language model trained at substantial scale to acquire broad language capabilities. Its computation uses stored numbers called parameters. During training, examples are used to adjust those numbers so that the model becomes better at its training task. When we later give it a question, it uses the resulting numbers to calculate a response. Chapter 2 will separate changing those stored numbers from changing the question. There is no single universal numerical threshold for “large.” Part I concentrates on causal text generation: extend the available sequence by selecting successive tokens. Later chapters distinguish model architectures and training objectives more carefully.

A probability distribution over continuations is useful for producing text, but it is not itself a current policy repository, an order service, or an authorization mechanism. The system contains the model together with the software that selects input, obtains evidence, retains records, executes permitted operations, and presents results. Two systems using the same model can therefore differ in both capability and reliability.

Request path

1. Application record1. Selected messages and approval state
2. An operation2. Application builds context
3. Stored model numbers3. Model uses fixed weights
4. An operation4. Application checks output

Data → 1 → 2 → 3 → 4; control ⇢ application decides execution.

External evidence

1. Outside the modeltravel-v2: 750 CNY/night
2. Outside the modelorder-v1: A-104 processing

External data → selected context (step 2). A file existing is not a read.

Figure 1.1 — The model is one computation inside the system. Original book diagram · CC BY-SA 4.0.

Figure 1.1 locates the responsibilities exposed by the policy-update and order-query cases. External documents enter the model’s context only after the application selects them. An order read returns evidence to the application. The model uses the supplied material to produce text or a proposed operation. The application remains responsible for enforcing the boundary between a proposal and an executed change.

This decomposition makes failures actionable. If the application supplies travel-v1, improving the model’s ability to read travel-v2 does not repair that run: v2 was absent. If a current source is supplied but its number is misreported, the failure lies farther along the path. If a cancellation bypasses approval, fluent wording cannot repair the missing control. Investigation begins by locating which required step failed.

The five cases also clarify what an LLM is not established to be by its conversational appearance. Correct completion does not demonstrate universal truthfulness. A response that uses a document does not demonstrate permanent parameter learning. A sentence about an action does not demonstrate execution. These conclusions follow from the missing evidence, without requiring a claim about whether the system “really understands.” More specific questions about computation will be more productive than treating that broad word as an explanation.

1.6 Build an evaluation record that another reader can inspect

1.6.1 Preserve the conditions, criterion, and outcome

We can now locate errors, but a second reader needs enough information to follow the same investigation. Consider the policy answer again. “It said 600” reports a reply but leaves out the decisive fact: was the current document supplied? “It received travel-v2, which specifies 750, and answered 600” makes the discrepancy inspectable. If we also explain that an employee might use the wrong reimbursement limit, we have connected the error to its possible consequence.

An evaluation case contains an input, the relevant operating conditions, and a criterion. Running or examining the case supplies an outcome. A failure record preserves the discrepancy when the criterion is not met. A risk record describes a possible consequence and a control intended to prevent or limit it. A risk can exist even when the particular recorded run causes no harm.

Table 1.3 — Criteria and risks across the five cases.

Case	Criterion	Problem in this example	Possible consequence and response
Factual completion	Name Paris as the capital	The second response names Lyon	A false statement may be accepted; check an independent reference
Open-ended completion	Accept both plausible endings	Neither response fails; exact-string grading would wrongly reject sofa	A valid answer may be rejected; specify the allowed variation
Clue substitution	Answer red for the red key and blue for the blue key	The blue-key story still receives Red.	The clue may not be used; check the actual input, then test placement
Policy updates	Give 750 CNY under the current policy; disclose when it cannot be verified	The first response gives the old limit, 600 CNY	An expense decision may use an old rule; check the source version and whether it was supplied
Order queries	Report the supplied status, processing; disclose a missing lookup; leave the order unchanged	The first response claims shipped without a lookup	A false status may mislead later action; verify first and require approval before changes

This comparison is a baseline: a fixed reference against which to compare changes. It is not a base model, which describes a stage in a model’s training history. These similarly named objects serve different purposes.

A useful failure report also preserves the full input and exact response. “The policy answer was wrong” loses the information needed to distinguish selecting an old source from misreading a new one. For an actual model experiment, retain the model and tokenizer versions, instructions and selected history, source versions, tool results, output-selection settings, and exact outputs. If the program uses a repeatable sequence of random choices, retain its starting setting, called a random seed, too. It helps repeat that choice sequence; it does not record the input or the rest of the system. Chapter 4 gives a complete example.

1.6.2 Understand what an aggregate score discards

Example 1.6 — Scoring the authored collection. Under the criteria above, factual completion has one passing response, open-ended completion two, clue substitution one, policy updates one, and order queries one. Six of the ten records pass. If an evaluator wrongly accepts only mat in the open-ended case, it reports five passes instead. The records have not changed; the score changed because the criterion changed.

Even the correctly computed six-out-of-ten result describes only these ten deliberately chosen records. It is not a measured model accuracy. The collection was constructed to illustrate particular behaviors, not sampled from a real workload. Combining its cases into a percentage would also conceal the difference between harmless wording variation and an unsupported order claim.

When comparing systems, the evaluation collection and criterion must stay interpretable. If we repair policy retrieval, the most direct initial question is whether the policy cases now satisfy their source requirements. A broad average can supplement such analysis, but cannot replace it. Later evaluation chapters will address representative sampling, repeated runs, and uncertainty; here the essential habit is to keep each score connected to the cases and decisions that produced it.

1.7 Summary

The chapter began with a chat window and ended with a set of distinct questions. Does the response satisfy the task’s criterion? Did it use the supplied clue? Did the current source reach the computation? Does a status claim agree with a record? Did an authorized operation actually occur?

These questions separate fluent generation from factual agreement, acceptable variation from an error, context availability from context use, and generated text from system action. The model contributes a learned computation over language; the surrounding system supplies information and controls effects. The next step is to open that computation far enough to follow one token. Chapter 2 asks what changes during generation and what would constitute a change to the model itself.

1.8 Exercises

1. Criterion design. A task says “Give two possible destinations for a holiday in France.” An evaluator accepts only Paris and Lyon. A response says Nice and Bordeaux. Identify the evaluator’s mistake and propose a criterion. What extra constraint would make the original exact answer appropriate?
2. Controlled comparison. A system answers the red-key story correctly. Design two follow-up comparisons: one tests whether the clue’s value matters; the other tests its position. State what stays fixed in each comparison.
3. Evidence and truth. Suppose the unverified shipped response in the order-query case happens to match the actual order. Which property has improved relative to the recorded case, and which task requirement remains unestablished?
4. Source tracing. travel-v2 exists externally. The application’s recorded input contains travel-v1, and the response says 600. Identify the first demonstrated failure in the evidence path. Would replacing the final number manually establish correct source selection?
5. Aggregation. Calculate the per-group and total pass counts in Table 1.3. Then explain why adding ten copies of the Paris. response from Example 1.1 would improve the aggregate proportion without demonstrating any new capability.
6. Action evidence. A log contains a proposed cancellation and the final message Cancelled, but no approval or tool result. List what remains to be established. Explain why repeating the final message cannot fill the gaps.
7. Failure report. Write the input condition, expected behavior, observed discrepancy, and risk for a system that receives travel-v2 but reports 600. How does this differ from the first record in the policy-update case?
8. Limits of observation. A model changes its answer after a document is added to the input. Give two questions about the system’s retained state that this observation alone cannot answer.

1.9 Solutions

1. The task permits multiple correct pairs. The evaluator should require two destinations in France that meet any stated holiday constraints, allowing appropriate wording and order. Exact repetition becomes appropriate if the instruction explicitly says to return Paris and Lyon character for character; that is a different task.
2. For the value test, replace red by blue while preserving all other text and settings; the required answer changes to blue. For the position test, keep the blue clue and the other sentences unchanged but move the clue next to the question; the required answer stays blue. The first tests dependence on content, the second sensitivity to placement. Neither alone identifies an internal mechanism.
3. The statement is now factually correct. Evidence that the system consulted the correct order record is still missing. If verified status is required, accidental agreement does not establish that the procedure met the task.
4. The new source was not selected into the recorded input. Replacing the displayed number can correct that output string, but leaves the selection defect intact and gives no reason to expect the next query to use the right document.
5. The counts are 1/2, 2/2, 1/2, 1/2, and 1/2, totaling 6/10. Adding ten copies of a passing record gives 16/20. This changes the weighting of already observed behavior; it supplies no new task or independent measurement and demonstrates no improvement in the system.
6. The application must establish permission, the required approval, execution of the cancellation, and a confirming result or state record. A proposed operation is not execution. Another generated message adds text to the log without supplying those events.
7. Input: the current travel-v2 excerpt is supplied with the current-limit question. Expected: 750 with the source identified. Discrepancy: the answer says 600 despite the new excerpt. Risk: an obsolete allowance may be used. Unlike the first record in the policy-update case, missing source selection is no longer the demonstrated defect; investigate use of the supplied evidence.
8. We do not know whether the application’s history will retain and reinsert the document in a later run, or whether any separate process changed model parameters. A changed response is compatible with changed input alone.

1.10 References

• Lin, Hilton, and Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022). The authors evaluate answers that repeat common human misconceptions. Read this alongside the factual-completion case to see why fluent language needs a separate truthfulness criterion.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023). This study compares model performance when relevant information appears at different positions in the context. It gives a research counterpart to the placement test proposed in the clue-substitution case.
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020). The paper combines information learned by a model with documents retrieved by surrounding software. This distinction helps explain why the current policy must reach the model before it can serve as evidence for an answer.
• Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models (2023). The method alternates generated reasoning and action proposals with observations returned by an environment. It helps distinguish proposing an order lookup from receiving its result; the approval rule in our example is a separate application requirement.