Chapter 2 The Journey of a Token: Training Is Not Inference

2.1 The problem: a changed answer has several possible causes

In Chapter 1, supplying travel-v2 changed the policy answer from 600 to 750. It is natural to describe this as learning the new policy. But the observation leaves an important question unanswered: did the system change the computation it had learned, or apply the same computation to different information?

A simple calculation already illustrates the distinction. A rule that takes an input and returns a result is called a function. For the rule “multiply the input by two,” input 3 produces 6 and input 4 produces 8. The output changes while the multiplier stays fixed. Replacing the multiplier with five changes the rule instead. A language model is not this arithmetic function, but it too has both supplied input and stored numerical values. We need to follow those objects separately.

Our central question is: which objects change when a model generates a response, and which additional operations change the model itself? The answer requires two views of the same computation. The first follows information through one prediction. The second follows the feedback after that prediction: either extend the input sequence or update learned parameters.

Keep the opening calculation in mind: the input is 3, the stored multiplier is 2, and the result is 6. In the model, these three roles become the supplied text, the stored parameters, and the calculated output. We will follow four operations: encode text, calculate probabilities, choose and append a piece, and—only in the training path—update parameters.

We will use a simplified causal Transformer text generator. Transformer is the name of a way to organize a neural model’s calculation; its individual components are not prerequisites for this chapter. “Causal” here means that a prediction is conditioned on earlier tokens, without access to the future tokens it is supposed to predict. This is a description of the dependency allowed in the sequence; it is not a claim that the model has discovered causes in the world. The original Transformer paper provides the architectural starting point. Our map isolates a causal generation path rather than reproducing that paper’s complete encoder-decoder architecture. Vaswani et al., 2017

2.2 Follow one prediction from text to numerical representations

2.2.1 Select the input and encode its pieces

The calculation inside the model operates on numbers, so text needs a numerical representation before it can enter. We do not begin by assigning one number to the meaning of an entire sentence. Instead, we split the text into reusable pieces and identify each piece.

Example 2.1 — Continuing a factual sentence. The visible text is The capital of France is. An application could add an instruction, earlier conversation, or a retrieved excerpt before sending it to the model. The complete selected material is the context for that run. For this example, assume that only the five-piece sentence below is supplied:

Text:    The capital of France is
Pieces:  [The] [␠capital] [␠of] [␠France] [␠is]
IDs:        1       2       3       4        5

Square brackets mark piece boundaries in the display; they are not characters added to the sentence. The ␠ symbol displays an ordinary space. A tokenizer applies an encoding scheme that turns text into a sequence of tokens and their identifiers. The possible entries form its vocabulary. Here ␠France is one piece and has ID 4. The identifier names an entry; it does not measure the meaning of France. Chapter 3 specifies this small tokenizer completely.

The numerical sequence preserves order. [1,2,3,4,5] is not a set of five unrelated numbers: changing their order changes the input. It also represents only the selected text. The encoding step cannot include a policy file that the application never supplied.

2.2.2 Look up numerical input, then compute with context

An ID tells us which piece we have, but its numerical size is arbitrary. Numbering France as 4 does not make it twice anything numbered 2. We therefore use the ID to look up a separate list of values intended for calculation. This is like using a row number to locate an entry in a table: the row number and the entry’s contents have different jobs.

The model uses each ID to retrieve a row from an embedding table. An embedding row is a vector: an ordered list of numbers used in the subsequent calculation. [2,1] means “first value 2, second value 1,” not 2 plus 1. Its two positions are called dimensions. Here we need only the idea of keeping several values together; Chapter 3 will draw such pairs as points. In our teaching table, ID 4 retrieves [2,1].

Table 2.1 — The three objects in one lookup.

Object	Value in the example	Role
Token piece	␠France	A segment of encoded text
Token ID	4	An address in a versioned vocabulary
Input embedding	[2,1]	Stored numerical values read at that address

No calculation turns the integer four into two and one. The lookup copies values that were already stored. The whole five-token input retrieves five such rows, in sequence order. Our two-number rows are hand assigned. Neural models usually learn their embedding values during training.

Lookup alone is not enough to choose a continuation. The piece France has the same entry in a question about a capital as in a question about an area, yet the expected answers differ. The model must also combine information from the surrounding pieces.

Transformer layers, successive stages of calculation, do this with the retrieved rows and information about position. They produce contextual representations: numerical results that can depend on the other available tokens, rather than only on one token’s vocabulary entry. The input lookup for ␠France is fixed under a fixed table; the subsequent representation is computed for this particular sequence.

At this stage, we specify the interface without yet deriving attention or matrix multiplication: ordered token representations and position information enter; context-dependent numerical representations leave. Replacing France in the input can change the computed results while all stored weights, another name for the learned numerical parameters, remain fixed. The mechanism of that interaction is the subject of later chapters. Its role here is to explain why a fixed model can respond differently to different contexts.

2.2.3 Produce scores, probabilities, and one selected ID

After the model has processed the available text, it must express how much it favors each possible next piece. Each possible choice is a candidate. The output calculation assigns a score to each candidate next-token ID. These raw scores are logits. A logit can be negative or greater than one. It is therefore not a probability. To use Chapter 1’s sampling rule, we need a share of the total chance for each candidate. A normalization calculation converts the scores into those shares: a probability distribution with nonnegative values whose total is one. Because the original scores can be negative, this conversion is not simply dividing every raw score by their sum. Chapter 6 derives that conversion.

To examine selection independently of that derivation, suppose the next possibilities are grouped into the following three illustrative labels. The probabilities are given, not calculated from our two-dimensional embeddings.

Table 2.2 — A specified distribution and its sampling intervals.

Candidate label	Probability	Assigned interval
Paris	0.7	0 inclusive to 0.7 exclusive
Lyon	0.2	0.7 inclusive to 0.9 exclusive
Other	0.1	0.9 inclusive to 1 exclusive

“Other” groups the remaining possibilities; it is not a token in our toy vocabulary. The table explains selection among these labels, not a complete real model vocabulary. Read [0.7,0.2,0.1] as 70%, 20%, and 10% of the selection chance. Starting at 0, allocate 0.7 to Paris; adding 0.2 puts the next boundary at 0.9; adding the final 0.1 reaches 1. This is where the interval endpoints come from. The intervals cover the whole range without overlap, and their lengths add to one.

The distribution still does not specify a unique output. Greedy selection chooses a largest-probability candidate and returns Paris here. Sampling uses a draw and the intervals: 0.8 selects Lyon. At exactly 0.7, the first interval has ended and the second has begun, so the result is also Lyon. The half-open boundaries make the rule unambiguous.

Prediction and selection are thus separate operations. The same distribution can produce different sampled outputs. Conversely, different distributions can have the same largest entry and therefore the same greedy output. Observing only the chosen word loses information about the computation that preceded it.

The value 0.7 expresses a next-choice probability under this construction. It is not a calibrated 70% probability that a factual claim is true. Chapter 1’s independent factual criterion still applies.

2.3 Close the generation loop

Suppose the selected piece is ␠Paris, ID 6 in our teaching vocabulary. The next step is to append that ID to the current sequence. The extended sequence becomes the input for the next prediction.

Table 2.3 — Two successive append operations in Example 2.1.

Prediction	Sequence available before selection	Selected ID	Sequence after append
First	[1,2,3,4,5]	6, ␠Paris	[1,2,3,4,5,6]
Second	[1,2,3,4,5,6]	7, .	[1,2,3,4,5,6,7]

The second row assumes the full stop was selected; it is another step, not a consequence numerically derived from Table 2.2. Each next distribution is computed for the sequence available at that step. Generating Paris therefore changes what the later calculation can condition on.

Input and representation

1. Text pieces1. Text → token pieces
2. Row number2. Token IDs in order
3. A list of numbers3. Embedding rows
4. An operation4. Transformer + position

Data → 1 → 2 → 3 → 4. Parameters supply the table and layer weights.

Prediction and selection

1. An operation5. Logits → probabilities
2. Row number6. Select one next ID
3. Application record7. Append ID to sequence
4. Text pieces8. Detokenize text

Data → 5 → 6 → 7 → 8; control ⇢ repeat from 3 with extended IDs, or stop.

Figure 2.1 — One selection in a causal text generator; ID append closes the loop. Original book diagram · CC BY-SA 4.0.

Figure 2.1 shows the return path from the appended ID to the numerical computation. This repeated prediction-and-append process is autoregressive generation: the sequence is extended using its own earlier tokens, including tokens already generated. An implementation can reuse intermediate work instead of recalculating every earlier position. That efficiency change does not alter the dependency on the available sequence; caching is deferred to Chapter 30.

Detokenization converts output IDs into readable text according to the matching encoding scheme. The interface may display several pieces together. Consequently, the chunks that appear on screen need not reveal token boundaries.

A real generator may stop when it selects a designated end token, when a configured stop condition is satisfied, or when a limit is reached. Our tiny Chapter 3 vocabulary illustrates the sentence encoding but contains no automatic end token. Chapter 4 introduces explicit beginning and ending conventions for a separate character model. In either setting, choosing to end and being interrupted at a length limit are different events.

We can now state an invariant of the ordinary generation loop: the sequence and intermediate results may change, while the model parameters stay fixed. Parameters include the embedding rows and learned layer weights. Computed intermediate values are often called activations. Reading a parameter into a calculation does not update that parameter, just as using a stored multiplier does not replace it.

2.4 Training feeds an error back to the parameters

2.4.1 A known sequence supplies many prediction targets

We have followed what happens when parameters are already available. Where did they come from? Return to the sentence that we just continued. To generate it, we had to choose the next piece without knowing the answer. To train on an existing copy of it, we can hide the next piece from the prediction and then use that known piece to check the result.

Training estimates or adjusts a model’s learned state from designated data. For the neural next-token model in this chapter, the training procedure uses known text to define prediction targets. Generation and training share a forward computation, but use its result differently.

Example 2.2 — Constructing the targets. Take the complete sentence The capital of France is Paris. Its IDs are [1,2,3,4,5,6,7]. A prefix is the initial part of a sequence: [1,2], for example, keeps its first two IDs. The next token after the prefix [1] is known to be 2; after [1,2] it is known to be 3, and so on.

Table 2.4 — Target alignment within one training example.

Prediction position	Allowed prefix	Target ID and piece
1	[1]	2, ␠capital
2	[1,2]	3, ␠of
3	[1,2,3]	4, ␠France
4	[1,2,3,4]	5, ␠is
5	[1,2,3,4,5]	6, ␠Paris
6	[1,2,3,4,5,6]	7, .

Read the fifth row in ordinary language: “Given The capital of France is, predict the piece ␠Paris.” The fourth row stops one piece earlier: “Given The capital of France, predict ␠is.” Moving the dividing line through one existing sentence gives us several prediction exercises. The targets are the original sequence shifted one position forward. This example evaluates six next-token targets; it does not define a start prediction or a target after the final full stop. The training program knows the full example, but each prediction must use only its allowed prefix. Otherwise, a position could read its own future answer and the task would no longer test next-token prediction.

Because the correct prefixes are already available in the training data, many positions can be processed together while enforcing those visibility restrictions. During ordinary generation, the next generated prefix does not exist until a token has been selected. The two procedures therefore need not have the same execution schedule, even though they use the same next-token task.

2.4.2 Separate measuring error, finding a direction, and updating

Knowing the target gives us a way to compare predictions. If the observed next piece is Paris, a prediction that gives Paris probability 0.1 fits that example less well than one that gives it 0.7. Choosing the most likely word alone would discard this difference in confidence; training can use the probability assigned to the target.

A forward calculation computes predictions using the current parameters. A loss assigns a numerical penalty to the disagreement between predictions and supplied targets. In next-token training, putting little probability on the observed next token incurs a larger penalty. The target comes from the training text; it is not automatically a certified statement about the world.

Measuring an error still does not tell the program what to change. In the opening multiplication example, suppose input 3 should produce target 9, but the stored multiplier 2 produces 6. Slightly increasing the multiplier moves the result toward 9; decreasing it moves the result away. With many parameters, training needs corresponding information about how small changes affect its loss.

A gradient describes that local sensitivity of the loss to changes in parameters. An optimizer uses gradient information and an update rule to change parameter values. These are three distinct jobs: measure mismatch, determine how the mismatch responds to small changes, and apply an update. Chapter 7 develops their mathematics. Naming them here specifies the additional steps that must exist before we can claim a neural training update occurred.

Training timeline

1. Text pieces1. Text + known targets
2. An operation2. Forward → loss → gradient
3. Stored model numbers3. Optimizer updates θ
4. Stored model numbers4. Save checkpoint

Data → 1 → 2; update ⇒ 2 ⇒ 3; control ⇢ repeat training, then save 4.

Inference timeline

1. Application record1. Selected context
2. Stored model numbers2. θ stays fixed
3. An operation3. Forward → probabilities
4. Row number4. Select and append ID

Data → 1 → 3 → 4; parameters 2 feed 3; control ⇢ append and repeat. No update ⇒.

Figure 2.2 — Data flow →, parameter update ⇒, and control ⇢ have different roles. Original book diagram · CC BY-SA 4.0.

In Figure 2.2, the training path feeds the loss-related information back into the stored parameters. The generation path feeds a selected ID back into the sequence. Both contain a forward calculation. Only the first contains the loss-based parameter update described here.

This difference has several consequences. Producing a wrong answer does not automatically train the model to correct it. Supplying a correction in the next message changes the next input; a parameter update would still require a separate training operation. Saving the conversation preserves text, not new weights. And a training update need not write a fact into one dedicated location: it can change many numerical relationships and affect other predictions too.

Training is broader than gradient-based neural training. Chapter 4 will fit a small model by counting observations. What makes that counting training is that it estimates the model’s state from designated data; the later act of reading those counts for a prediction does not fit them again.

A saved model state is a checkpoint. To execute it correctly, the parameters must be used with the corresponding architecture and compatible tokenizer and ID mappings. A checkpoint for resuming training may also contain optimizer and other training state. A chat transcript alone is neither kind of checkpoint.

2.5 Separate the four places where information can change

2.5.1 Define the objects before comparing runs

We can now analyze the policy example more precisely. An application can change several kinds of information, and “the system remembered” does not say which one was involved.

Table 2.5 — Four locations and their update operations.

Location	Example contents	Operation that changes it
Model parameters	Embedding rows and learned layer weights	Apply a training update or load another checkpoint
Current context	Question, selected messages, and policy excerpt	Construct a different input
External data	Policy document outside the model	Edit or replace that document
Application state	Saved history or approval record	Write an application record

Loading a different checkpoint changes which parameters are used; it is not the training procedure that produced them. Similarly, saving a new document and selecting it into an input are separate operations. A file can change while an application continues sending its old contents.

We can give this distinction a compact mathematical notation. Read the equation below as one instruction: “use the stored parameters to process the supplied sequence and return probabilities.” The letters name objects we have already encountered. Let $s$ denote the selected token sequence and $\theta$ the parameters, with theta serving as a compact name for all the stored model numbers. Write the distribution produced by a fixed computation as:

$$p=F_\theta(s).$$

Here $p$ is the vector of next-token probabilities, and $F$ names the computation. The parentheses indicate what is supplied to the computation; the subscript theta indicates which stored parameter values it uses. The equation packages the relationship, rather than deriving the model’s internals. In the simplified deterministic forward computation considered here, equal $s$ and equal $\theta$ give equal $p$. A file or saved record can influence this computation only if the application uses it to change one of the relevant inputs. The later selection rule can still choose different tokens from the same distribution.

2.5.2 Analyze six fully specified states

Example 2.3 — A finite stand-in for the dependency. To isolate the four locations without running a neural network, define a table-based function with two whole-answer labels, 600 and 750. They are answer labels, not tokenizer IDs. Parameter version A and parameter version B select the following explicitly assigned tables.

Table 2.6 — The complete function, in answer order [600, 750].

Parameter version	Selected travel-v1	Selected travel-v2
A	[0.8,0.2]	[0.1,0.9]
B	[0.3,0.7]	[0.05,0.95]

These numbers are assumptions of a teaching function. They are not learned weights, not outputs derived from the preceding embedding rows, and not estimates of a real model’s accuracy. Their purpose is to make the dependency on selected context and parameter version completely inspectable.

For example, with parameter version A and v2 in the input, find row A and column v2: [0.1,0.9] gives 600 a 10% chance and 750 a 90% chance. Notice that no step asks what is merely stored in an external file. The next table changes those storage conditions so we can see when they do and do not reach the calculation.

Table 2.7 — All six fixed scenarios and their results.

Scenario	Parameters	Context sent	External file	Saved history	Output [600, 750]
Initial state	A	v1	v1	None	[0.8,0.2]
File replaced	A	v1	v2	None	[0.8,0.2]
New document supplied	A	v2	v2	None	[0.1,0.9]
History saved	A	v1	v1	v2	[0.8,0.2]
History supplied	A	v2	v1	v2	[0.1,0.9]
Parameters replaced	B	v1	v1	None	[0.3,0.7]

Compare “Initial state” with “File replaced”. The external file changes, but the table lookup still receives A and v1. The result must therefore remain [0.8,0.2]. Comparing “File replaced” with “New document supplied” changes the selected context to v2, so the lookup moves to the other column. Parameters A remain unchanged.

The same reasoning explains “History saved” and “History supplied”. Saving v2 in history has no effect while the application sends v1. Selecting that saved text changes the context and therefore the output. A persistent-looking memory can thus be implemented by saving and reinserting text, without changing parameters.

Finally, “Parameters replaced” changes the table version while preserving the old selected input. Its distribution differs, even though the visible question and source can stay the same. Under greedy selection, both this case and “New document supplied” choose 750. The identical answer conceals different causes: a changed parameter version in one case and a changed context in the other.

All six scenarios are fixed records in part-i-v1. In each row, identify the parameter version and the context actually sent, then look up their intersection in Table 2.6. Real retrieval systems are much richer, but use a related distinction between learned state and externally supplied information. Lewis et al., 2020

2.6 What adaptation and reasoning do—and do not—establish

The six cases let us distinguish changes in the system. Two common uses of the word “learning” or “thinking” can otherwise obscure that distinction. In particular, generating the steps of a solution does not by itself add a parameter update to the generation loop.

Inference means applying a model to an input; we also call it model execution or a model run. Reasoning means working through a problem, such as connecting premises to a conclusion. A model can generate reasoning during inference while its parameters remain fixed. The two words describe different dimensions of the activity.

Examples in a prompt can change how a fixed model responds. For instance, demonstrations can specify an unfamiliar answer format. This behavior is commonly called in-context learning. The name describes adaptation through supplied context; it does not imply a task-time gradient update. Brown and colleagues studied few-shot task use without such updates. Its success on a particular task still requires evaluation. Brown et al., 2020

A system may also collect feedback for later training or explicitly adapt online. These are identifiable additional processes. To distinguish them from in-context adaptation, inspect which objects were stored and changed, rather than infer an update from a helpful response.

Generated reasoning raises a separate evidential question. A chain of explanatory sentences is output text. Its steps may be valid and useful, but the text is not guaranteed to report every internal cause of the answer. Research has demonstrated explanations that omit factors influencing a model’s response. This does not make all explanations false; it means their logical and factual adequacy must be evaluated independently of any claim to reveal the complete computation. Turpin et al., 2023

For the policy case, a good explanation should identify the applicable source and correctly connect its contents to the answer. That can justify the answer to a reader. It still does not demonstrate a new checkpoint, an automatic training update, or a faithful internal trace.

2.7 Summary

One generated token passes through several distinct representations: text pieces, IDs, embedding rows, contextual numerical results, scores, probabilities, and a selected ID. Appending the ID changes the sequence on which the next prediction depends. Ordinary generation uses fixed parameters throughout that loop.

Neural training adds supplied targets, a loss, gradients, and an optimizer update. Its feedback changes parameters rather than merely extending a response. At the system level, parameters, current context, external data, and saved application state can change independently. Table 2.7 shows why neither a changed answer nor an unchanged answer uniquely identifies which object changed.

Chapter 3 now makes the encoding interface executable by hand: how the pieces are selected, how an ID retrieves a row, and which properties a comparison between two rows actually measures.

2.8 Exercises

1. Sampling boundaries. Apply Table 2.2 to draws 0.699, 0.7, 0.899, and 0.9. Which would differ from greedy selection? Explain why a rule that includes 0.7 in both adjacent intervals is incomplete.
2. State tracking. After the first append in Table 2.3, identify one object that has grown, two stored objects that have not changed, and one computed object that may change on the next pass.
3. Target alignment. For the four-token sequence [8,9,10,11], write all next-token prefix/target pairs under the same convention as Example 2.2. Why must the first prediction not read ID 9 from its input?
4. Necessary versus sufficient evidence. An application logs a forward calculation and a loss value. Has it demonstrated a parameter update? State the missing operations or evidence.
5. Function lookup. A run uses parameter version B and selects v2 from saved history while the external file still contains v1. Read its output from Table 2.6. Which field of the external file must this table-based function inspect to obtain that result?
6. Same answer, different process. Find two rows of Table 2.7 that greedily answer 750 for different reasons. What additional records distinguish them?
7. Counterexample to permanent learning. Design a two-conversation procedure that could reveal that an apparent policy memory depends on reinserting history. What would the result still not prove about every other memory mechanism?
8. Model run and reasoning. An assistant emits a valid multistep calculation and the application saves the transcript. Classify the generated reasoning, the model run, and the saved transcript. Which event establishes a new checkpoint?

2.9 Solutions

1. The selections are Paris, Lyon, Lyon, and Other. All except the first differ from greedy Paris. If 0.7 belongs to both intervals, a draw at that boundary has two possible answers under the stated rule. Half-open intervals assign it to exactly one candidate.
2. The current ID sequence grows from five entries to six. The vocabulary and embedding table remain fixed. The next contextual representation or next-token distribution may change because the sequence now includes Paris. A different computed result is not a changed parameter.
3. The pairs are [8]→9, [8,9]→10, and [8,9,10]→11. There are three predictions. Allowing the first calculation to read its target ID 9 would leak the answer, so it would no longer test prediction from the earlier prefix.
4. No. A loss can be computed for evaluation without changing weights. Neural training also needs the relevant gradient calculation and an applied optimizer update. To establish that parameters changed, retain the update/state evidence; merely naming or calculating an error does not establish that it was applied.
5. The output is [0.05,0.95]. No external-file field is read by the defined table lookup; its arguments are B and selected v2. The application already made the selection. Changing an unused external file cannot alter this function result.
6. “New document supplied” uses A with new context v2; “Parameters replaced” uses B with old context v1. Both choose 750 greedily. Retaining the parameter/checkpoint identity and the actual selected input distinguishes the two paths, as do their full distributions.
7. Supply v2 in one conversation, then start a second while recording whether saved text is reinserted. Compare an input including that history with an otherwise identical input omitting it. If only the first yields the current answer reliably under controlled settings, that supports dependence on supplied history in this setup. It does not prove the absence of parameter knowledge or every other storage mechanism; those require separate evidence.
8. The multistep output is reasoning to assess. Computing it by applying the model is inference. Saving it changes application state. None of these events, as stated, saves new learned model parameters, so none establishes a new checkpoint.

2.10 References

• Vaswani et al., Attention Is All You Need (2017). This paper introduces the Transformer architecture. Our token-journey diagram provides an initial map of a causal text generator; later chapters explain the layer operations in detail.
• Brown et al., Language Models are Few-Shot Learners (2020). The study supplies task demonstrations in the input without updating model parameters for the task. Compare this setup with the chapter’s distinction between changing context and changing learned parameters.
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020). The architecture combines learned model information with retrieved documents. It provides a concrete setting in which an answer can use new evidence without that evidence first becoming a parameter update.
• Turpin et al., Language Models Don’t Always Say What They Think (2023). The experiments show that generated explanations can fail to reveal factors that influenced an answer. Read this alongside the discussion of why visible reasoning text is not a direct trace of internal computation.