Chapter 34 SFT: From Completion to Instruction Following

34.1 The next word may be the wrong kind of answer

“The capital of France is” invites a continuation. “Name the capital of France; answer briefly” asks for a particular kind of response. A model could continue the second string with another instruction, a quoted conversation, or a long travel description. Fluent continuation alone does not establish that it will answer the request.

Chapter 22 separated a training objective from a model’s architecture. We keep the causal decoder and its next-token predictions. What changes here is the material those predictions learn from: examples of a request followed by a response that we intend an assistant to give. Training on these demonstrations is supervised fine-tuning (SFT). “Supervised” means that the desired output is provided; it does not mean that somebody watches every update.

Success has two levels. First, the record must score the intended answer tokens under the intended conversation prefix. Second, an adapted model must follow requests beyond its demonstrations while retaining behaviors we still need. A perfectly formatted dataset establishes the first level only. KA-1 makes this distinction concrete: its English 750. can convey a correct current amount, yet the identical response to a historical question is wrong.

We will make one record inspectable by hand. Chapter 18 supplies shifted targets, and Chapter 6 supplies negative log likelihood. The optional CPU path uses the existing Python primer. No GPU or model account is needed.

34.2 A conversation has to become one sequence

Example 34.1 — Serialize a short answer. Our demonstration has three messages: system content brief, user content France?, and assistant content Paris .. Here the first two abbreviate “answer briefly” and the capital question. These spellings and the tokenizer below are author-assigned teaching values, separate from MG-2 and real chat formats.

A role identifies who supplied a message. A chat template specifies how messages become tokens. Ours writes one beginning marker, then each message’s role marker, content and end-of-turn marker. A marker is a token with an agreed use, not an independent authority-enforcement program.

Table 34.1 — The complete toy vocabulary. Array order defines each ID. An ID is a label, not a magnitude.

Token	ID	Use
<bos>	0	Start the sequence once
<system>	1	Start the system message
brief	2	System content
<eot>	3	End a message
<user>	4	Start the user message
France?	5	User content
<assistant>	6	Start the answer
Paris	7	First answer token
.	8	Second answer token
<pad>	9	Unused padding label

Applying the template gives eleven tokens. Spaces below separate tokens for inspection; they introduce no additional IDs.

<bos> <system> brief <eot> <user> France? <eot> <assistant> Paris . <eot>
0     1        2     3     4      5       3     6           7     8 3

The prefix through <assistant> is what we supply when asking this toy model to begin an answer. During training the correct answer is present too, so later predictions can use earlier correct answer tokens. This is the teacher-forcing arrangement used in earlier next-token training. A causal attention mask prevents the prediction of Paris from reading that target or the later period in advance.

Actual templates differ in roles, delimiters and termination. Some assistant prefixes contain several tokens; some formats need no separate assistant prefix. A template and tokenizer belong with the checkpoint that learned them. The dated official template documentation also explains why adding special tokens twice changes the sequence. Our toy rule is a complete local specification, not a format to impose on another model.

34.3 Which prediction pays for the mistake?

The model reads a prefix ending at one position and predicts the next token. We choose to score assistant content and its terminating <eot>. We do not ask this exercise to learn to generate system or user messages. A loss mask is a list of zeros and ones selecting which predictions contribute to loss. It differs from an attention mask: an unscored prompt token can still inform an answer.

Table 34.2 — Every position in Example 34.1. Positions start at zero. The target is one token to the right. A dash means there is no target.

Input position	Input token / ID	Next target / ID	Loss mask	Why
0	<bos> / 0	<system> / 1	0	Role header
1	<system> / 1	brief / 2	0	System content
2	brief / 2	<eot> / 3	0	System end
3	<eot> / 3	<user> / 4	0	Role header
4	<user> / 4	France? / 5	0	User content
5	France? / 5	<eot> / 3	0	User end
6	<eot> / 3	<assistant> / 6	0	Answer header is supplied
7	<assistant> / 6	Paris / 7	1	Learn the first answer token
8	Paris / 7	. / 8	1	Learn the continuation
9	. / 8	<eot> / 3	1	Learn to end the turn
10	<eot> / 3	—	—	No following token

Read position 7 carefully. Its input is a role marker, but its prediction is the first answer token. Masking by the input token’s role would discard this signal. The mask belongs to the target being predicted. The ten masks are 0 0 0 0 0 0 0 1 1 1.

Example 34.2 — Compute the answer loss. Assign probabilities 0.5, 0.25 and 0.5 to the three correct targets in order. Each conditions on a different prefix; they are not alternatives in one distribution. Their negative natural logarithms are approximately 0.693147, 1.386294 and 0.693147 nats. Add the costs and divide by three scored predictions:

$$L=\frac{-\ln(0.5)-\ln(0.25)-\ln(0.5)}{3} =\frac{\ln(16)}{3}=0.924196.$$

The denominator is 3, not eleven stored tokens or ten available predictions. The period contributes twice either other cost. Raising its assigned probability to 0.5 lowers the mean to 0.693147. This is an arithmetic relationship under assigned probabilities, not an observed training improvement.

For a general record $x_0,\ldots,x_T$, let $m_t$ select the prediction made after reading through position $t$. Let $p_\theta(x_{t+1}\mid x_0,\ldots,x_t)$ be the correct next token’s probability under parameters $\theta$. Our selected-target mean is

$$L(\theta)=\frac{\sum_{t=0}^{T-1}m_t\left[-\ln p_\theta(x_{t+1}\mid x_0,\ldots,x_t)\right]}{\sum_{t=0}^{T-1}m_t}.$$

Here $T=10$, and only $m_7,m_8,m_9$ equal one. Substitution recovers Example 34.2. A record with no scored targets has a zero denominator; reject it or explicitly exclude it before averaging. Assigning it loss zero would describe missing training data as perfect prediction. Across a batch, add selected costs and selected-token counts before dividing. Averaging record means instead weights each record equally, a different possible objective that must be declared.

First answer prediction

1. Text piecesPrefix: … France? <eot> <assistant>
2. An operationInput position 7 → target Paris; m₇ = 1
3. An operation−ln(1/2) = 0.693147 nats

The header is supplied, but the answer predicted after it is scored.

Selected loss and update

1. An operationParis, period, end: 3 selected targets
2. An operationln(16) / 3 = 0.924196
3. Stored model numbersShared parameters may change; prompt loss = 0 does not freeze them

Loss selection and the choice of trainable parameters are separate decisions.

Figure 34.1 — The loss mask follows the target, not the input role. Original teaching diagram and assigned probabilities · CC BY-SA 4.0

Figure 34.1 connects the prefix, the first scored target and shared parameters under this loss mask. A zero loss bit does not freeze parameters used to process that position. Answer loss can flow backward through computations that used France?. Which parameters the optimizer may change is a separate choice. The loss mask also gives no permission to read future answer tokens; causal masking still applies.

34.4 A valid-looking record can teach the wrong task

Remove <assistant> from the runtime prefix. Its last token is now the user’s <eot>, not the header after which Paris was scored. We are asking for a distribution under a different condition. More training on correctly templated records does not repair this serialization mismatch. Inspect actual IDs on both paths first.

In a multi-turn record, inspect every answer span and terminator. “Score everything after the first assistant marker” also scores later user messages. Adding two padding positions creates no new answer targets: their loss bits remain zero, and padding must not become useful context. Truncating a long record can remove its answer or terminator. Silently accepting the fragment changes what the model is taught. Packing dialogues also requires explicit attention and boundary rules.

Our checker serializes the structured messages and compares the sequence and shifted mask to the declared rule. Its fixed inputs are in data/part-vi/sft-v1.json; code/part-vi/core.py implements it. The outcomes are visible below.

Table 34.3 — Format audit under the toy contract. These deterministic outcomes are not model-quality scores.

Record	Outcome	Explanation
Complete three-message dialogue	Accept	11 tokens, 10 predictions, 3 scored targets
User role changed to customer	Reject	Unsupported role sequence
Empty assistant content	Reject	A demonstrated answer is required
Last <eot> removed	Reject	Incomplete serialized record
Mask shifted one position right	Reject	First answer prediction is lost
Control marker inserted as content	Reject	Content impersonates a template boundary

A real tokenizer may encode literal marker-like text differently. This exercise rejects it rather than implementing an escaping protocol. The narrow checker cannot certify a production format. Its useful invariant is agreement between record, template, tokenizer, shift and mask.

34.5 Good structure still needs good demonstrations

Suppose a lodging example answers “current limit: 600” despite supplied evidence of the newer 750 version. Its file can pass all the checks above. Training then rewards a stale answer. Another example might say “Cancelled” after receiving only an order status. That teaches an unsupported action claim, even with a correct assistant role and terminator.

Review demonstrations against their evidence, not merely a preferred tone. Pair current and historical questions so “always say 750” is not a shortcut. Include missing information, ungranted approvals, both languages and varying requests for brevity. Resolve conflicting labels and record origins and rights. SFT can train on worked solutions too, but a polished wrong solution is not a good target just because it is long.

Parameter updates can improve demonstrated behavior while worsening other behavior. Forgetting means a measured loss of previously useful performance after adaptation; it need not mean every old fact was erased. A narrow data mix can repeatedly move shared parameters toward one format or domain. Freeze a retention set—France, red/blue key pairs, old/current limits and approval boundaries—before selecting the training configuration. Inspect categories and languages separately. Fewer updates, a balanced mix or a smaller update are hypotheses to test, not guarantees.

InstructGPT illustrates demonstration training followed by preference stages, with evaluation and tradeoffs rather than a reliability guarantee. Ouyang et al. provide a reason to measure behaviors directly instead of selecting solely by training loss.

SFT names demonstrations and their objective. Full updates and LoRA name ways to change parameters. A job can be “SFT using LoRA” or “SFT using full updates.” Chapter 39 will calculate that distinction. First we need to decide what to do when two plausible answers disagree about which response is better.

34.6 Summary

A causal model can learn instruction following while continuing to predict the next token. The template supplies conversation structure; the shifted loss mask selects answer predictions. Inspect the first answer token, terminator and selected-token denominator explicitly. Structure, demonstration quality and retention are separate questions. SFT specifies neither the number of trainable parameters nor a guarantee of truthful answers.

34.7 Exercises

1. Keep Example 34.2’s first two probabilities but raise the end-marker probability from 0.5 to 1. Compute the mean loss. Does this show that the capital answer became more likely?
2. A second record has one scored target with cost 1.386294 nats. Combine it with Example 34.2 using a mean over scored tokens. Compare with an equal-weight mean over records.
3. A programmer scores positions whose input token is assistant content. Which scored prediction in Table 34.2 disappears? Does this prevent reading the user question?
4. Training contains only current-limit questions answered with 750. Design two tests distinguishing use of version evidence from repetition of 750. State the required answers.
5. A job freezes base matrices and trains low-rank factors on desired responses. Is it SFT, LoRA, both or neither? What evidence would establish improvement in the bilingual assistant?

34.8 Reference answers

1. The end cost becomes zero. The mean is $(\ln 2+\ln 4)/3=\ln(8)/3=0.693147$. Only termination changed; the probability of Paris remains 0.5. A lower total loss need not improve every component.
2. Add $\ln(16)$ and $\ln(4)$, then divide by $3+1=4$: $\ln(64)/4=1.039721$. The record mean is $(0.924196+1.386294)/2=1.155245$. It gives a one-token answer the same weight as a three-token answer.
3. The prediction of Paris after position 7 disappears because its input is a role marker. The period and terminator may still be scored, teaching how to finish without training the first content prediction. Loss selection does not remove user tokens from context.
4. Ask for the historical amount with old=600 and current=750; require 600. Then assign a new fictional version with current=800 and old=750; require 800. Keep evidence and scope explicit in both languages. These contrasts still do not replace evaluation on new document families.
5. Both: SFT describes demonstrations/objective, LoRA the factors being trained. Compare the unchanged contract before and after, including Chinese, historical scope, unknowns, approvals and retention tasks. Neither falling training loss nor a small adapter file establishes this improvement.

34.9 References

• Ouyang et al., Training language models to follow instructions with human feedback, 2022, sections 3.1, 3.5 and 5. Demonstration training, subsequent preference stages, evaluation and limitations in the InstructGPT setting.
• Hugging Face, Chat templates, Transformers v5.17.0. Serialization, generation prefixes and duplicate special tokens; checked 2026-09-13, separate from the dependency-free toy implementation.