Chapter 36 Training Reasoning Capabilities: Supervision, RLVR, and Verifiers

36.1 A number we can check without voting

Two nights cost 680 and 820 yuan. Our fictional rule reimburses each night up to 750 yuan; unused allowance cannot move between nights. What total may be reimbursed? We do not need to vote on which answer sounds more persuasive. The supplied rule determines the result.

This problem adds a rule to the continuing lodging example. It does not claim that any real employer uses it. The 750-yuan cap is provided in the question, so the task tests applying the rule, not remembering a current policy. If the cap were absent, calculation alone could not recover it.

Chapter 35 used preference labels. Here we examine feedback from checkable tasks, while retaining Chapter 32’s insistence on a fixed evaluation contract. Reasoning means problem-solving reasoning: combining given information and rules to reach an answer. It differs from inference, meaning running a model. Training problem-solving behavior can change parameters; checking a response at runtime need not.

36.2 The same final number can conceal different work

Example 36.1 — Apply the rule one night at a time. All costs and caps are whole yuan. For the first night, 680 is below 750, so reimburse 680. For the second, 820 exceeds 750, so reimburse 750. Add the two allowed amounts: $680+750=1430$ yuan.

The operation $\min(a,b)$ returns the smaller of two numbers. It lets us write the rule after doing the calculation:

$$R=\sum_{i=1}^{n}\min(c_i,h).$$

Here $n$ is the number of nights, $c_i$ the cost of night $i$, $h$ the per-night cap and $R$ the total allowed reimbursement, all amounts in yuan. For $n=2$, $c_1=680$, $c_2=820$, $h=750$, this is $\min(680,750)+\min(820,750)=1430$.

Now try adding the costs first: $680+820=1500$. Capping that total at $2\times750=1500$ gives 1500. That procedure transfers the first night’s unused 70 yuan to the second night, contrary to the rule. The arithmetic is internally consistent; the chosen operation is wrong.

Table 36.1 — Six authored candidates for the same problem. The allowed output is a whole-yuan amount alone, with ordinary grouping commas permitted. Any displayed working must also be valid if we claim a correct solution process. Candidate order is A–F.

Candidate	Claimed nightly amounts	Final response	Amount-output contract	Displayed work
A	680, 750; sum 1430	1430	Correct	Correct
B	680, 820; sum 1500	1500	Wrong	Second night is uncapped
C	700, 730; sum 1430	1430	Correct	Invented amounts; lucky total
D	680, 750; sum 1430	1,430	Correct	Correct
E	680, 820; sum 1500	Not 1430; pay 1500.	Wrong	Wrong rule and contradictory answer
F	680, 750; sum 1430	<answer>1430</answer> Cancelled A104.	Wrong	Arithmetic is correct; action claim is unsupported

A and C share a final number, but C is not a valid derivation. D changes formatting without changing the amount. F contains the correct number and correct arithmetic, yet makes a claim no reimbursement calculation can authorize. These distinctions let us test what a checker really checks.

36.3 Supervision describes the feedback, not just the model name

Outcome supervision labels a final result. A training record might mark A’s and C’s amounts correct, and B’s wrong. This supplies a useful signal, but cannot tell the learner where B went wrong or that C’s derivation is invalid.

Process supervision supplies feedback about intermediate steps. For B, the first allowed amount 680 is correct, while the second allowed amount 820 violates the cap. An annotation policy can stop at that first wrong step and leave subsequent steps ungraded. For C, the very first amount 700 is wrong. This requires more specific judgments, including rules for ambiguous or alternative valid steps.

Those labels can train a verifier: a program or model that checks a proposed answer or solution. A verifier’s inputs must include whatever it needs to check the claim. Our deterministic checker receives the nightly costs, cap and final response. It outputs acceptance or rejection and a reason. A learned verifier instead predicts a score from patterns in labeled examples; its score is evidence about likely correctness, not a proof.

There are three separate operations. A generator produces a solution. A verifier evaluates one. A training loop may use the feedback to change generator parameters. Checking a batch and returning the best answer performs the first two only.

Generate and check

1. Outside the modelGiven: costs 680, 820; cap 750 per night
2. Text piecesCandidate amount: 1430
3. An operationVerifier: min(680,750) + min(820,750) = 1430

An amount check accepts A and lucky C. It has not verified C's steps.

Two uses of acceptance

1. An operationRuntime: select a candidate; no update
2. Outside the modelTraining: accepted amount → assigned reward 1
3. Stored model numbersRL optimizer may update generator parameters

Supervised step/outcome labels can train a verifier separately. Neither checking nor selecting alone trains the generator.

Figure 36.1 — Checking a result becomes training only when feedback changes parameters. Original rule and process diagram · CC BY-SA 4.0

Figure 36.1 makes that separation explicit. In Lightman and colleagues’ process-versus-outcome study, the main comparison trains reward models and uses them to select among a fixed generator’s solutions. It does not train that generator with RL. Its strongest large-scale comparison also has differences between the training datasets; controlled smaller comparisons help investigate those differences. This supports a carefully scoped result, not a universal rule that step labels always win. The paper’s methods specify these conditions.

Supervised solution training provides another path: take a checked demonstration such as A and apply SFT to its answer and, when desired, its displayed steps. That trains the generator to predict a demonstrated sequence. It differs from training a verifier to label the same sequence. Inspect the training targets before inferring the job from a model’s name.

36.4 A checker can be wrong in two directions

Example 36.2 — Compare three small verifiers. V1 accepts if the raw response contains the characters 1430. V2 accepts only the literal string 1430 after surrounding whitespace is removed. V3 accepts only a well-formed nonnegative integer amount, with optional correctly grouped commas, then compares its value with the amount recomputed from the supplied costs and cap. V3 does not accept extra prose, tags or actions.

V3’s permitted comma form uses groups of three digits after the first group: 1,430 is accepted but 14,30 is malformed. This is a narrow numeric grammar, not a natural-language fact checker. None of these three verifiers inspects the claimed nightly steps.

Table 36.2 — Acceptance is not the same as correctness.

Candidate	V1: substring	V2: exact digits	V3: parsed amount and rule
A	Accept	Accept	Accept
B	Reject	Reject	Reject
C	Accept	Accept	Accept
D	Reject	Reject	Accept
E	Accept	Reject	Reject
F	Accept	Reject	Reject

Relative to the amount-output contract in Table 36.1, V1 accepts two bad responses, E and F. Those are false acceptances, also called false positives. It rejects correct D: a false rejection, or false negative. V2 eliminates those false acceptances but retains D’s false rejection. V3 fixes both kinds on this six-case set.

Yet V3 still accepts C. Under an amount-only contract that is a true acceptance; under a contract requiring valid working it is insufficient evidence. Changing the claim being checked changes the interpretation. To check the process too, compare every claimed nightly amount with its own cap calculation and verify the sum, while retaining the output grammar. This added check rejects C. Its success on these exact arithmetic fields does not make it a general proof checker for prose.

Format gaming exploits what the parser rewards. Adding an answer tag or a required suffix can satisfy a format reward while leaving the result wrong. A grammar check tells us whether the answer has the required structure. A numeric comparison tells us whether it matches this rule’s result. Neither alone verifies every statement in a longer response.

36.5 Use a checkable reward to change the generator

With a task whose result can be checked, assign reward 1 to an accepted amount and 0 otherwise. A loop can sample solutions, compute these rewards and update the policy. This is commonly called reinforcement learning with verifiable rewards (RLVR). The name describes the source of reward. It does not specify one optimizer, one architecture or a guarantee that the verifier is sound.

Example 36.3 — Feedback needs alternatives to distinguish. Suppose a sampled group gets rewards [0, 1, 0, 1]. The mean is $2/4=0.5$. Subtract it to obtain [−0.5, 0.5, −0.5, 0.5]: the second and fourth responses did better than their group. These centered rewards illustrate relative feedback; they are not a full PPO or GRPO implementation. If all four rewards are zero, this comparison does not identify any successful response to favor. Other training terms may still act, but the group has supplied no successful example.

Exploration means trying alternative outputs that can reveal such differences. It costs generation and checking work and may find more wrong answers. Under the strong assumption of independent attempts, each succeeding with probability $p=0.25$, four attempts contain at least one success with probability

$$1-(1-p)^4=1-0.75^4=0.683594.$$

We subtract the probability that all four fail from one. If the four attempts are exact duplicates, the probability stays 0.25; the independence calculation no longer applies. Moreover, containing a correct answer is not the same as selecting it. A bad verifier can discard the only correct candidate.

DeepSeek-R1 is one documented recipe, not the definition of reasoning training. Its R1-Zero stage starts from a pretrained base and uses rule-based accuracy and format rewards. R1 adds cold-start demonstrations and further training stages. Its reported distilled students use SFT on curated outputs without a student RL stage. “Without initial SFT” in R1-Zero does not mean without pretraining, and a format reward is not a correctness reward. The original report distinguishes these paths.

36.6 What would show that learning transfers?

Distillation transfers behavior through a teacher’s outputs or distributions. In the sequence-based case here, a teacher produces candidate solutions, a declared filter selects demonstrations, and a student learns them with SFT. This can move work from repeated teacher generation into student parameters. It also transfers selection bias, mistakes and gaps in the teacher’s coverage. Record the teacher, prompts, filtering rule, data rights and student evaluation, rather than calling every generated trace trustworthy.

A higher training-set score does not establish new-problem performance. If the exact 680/820/750 problem appears in both training and testing, memorizing 1430 suffices. Putting its Chinese translation in the test set is not an independent split. A near-copy with names changed can preserve the same shortcut. Group translations, paraphrases and document/problem families before splitting.

Changing the cap to 700 produces $680+700=1380$. That probes sensitivity to an input value. It does not by itself test a new rule family, such as a total-trip cap. Report these levels separately: new values within a known rule, new compositions of known rules, and genuinely different task distributions. Inspect verifier errors on the new distribution too; the judge can shift along with the generator.

Finally, an explanation visible on the page is generated text. Correct steps show that this text is a valid solution under the checked rules, not that it exhaustively records the model’s internal computation. Counterfactual experiments have found explanations omitting prompt influences that changed the answer. Turpin et al. establish that failure in particular model/task settings, not that all explanations are useless or unfaithful.

The CPU practice classifies every candidate using the frozen grammar and rules in data/part-vi/reasoning-v1.json. It also changes the cap and rejects malformed amounts. These are actual program checks over authored data, not a trained reasoning model benchmark. The next chapter keeps parameters fixed and asks how to spend extra generation and checking effort.

36.7 Summary

SFT on checked solutions, process/outcome supervision of verifiers, and RLVR of a generator are distinct jobs. A verifiable reward is only as good as its task specification and checker. Final-answer correctness can coexist with invalid working, and format compliance can coexist with a wrong answer. Exploration, distillation and generalization each need their own evidence. Visible reasoning text is not an internal execution log.

36.8 Exercises

1. The cap becomes 700 while costs remain 680 and 820. Compute the valid total and the incorrectly pooled total. Explain the difference.
2. Under the amount-output contract, count V1’s true acceptances, false acceptances, true rejections and false rejections across all six candidates. Which counts change for V3?
3. A training report calls C a “verified reasoning trace” because V3 accepts it. Locate the unsupported inference and specify an additional check.
4. Four duplicated solutions all fail. A report nevertheless uses the independent-attempt formula with $p=0.25$. Diagnose both the probability assumption and the distinction between pool success and selected success.
5. A student scores perfectly on the English originals and their Chinese translations used during distillation. Propose a split and evaluation that could support a narrower, defensible generalization claim.

36.9 Reference answers

1. Per-night reimbursement is $680+700=1380$. Pooling gives $\min(1500,1400)=1400$, incorrectly transferring 20 unused yuan. The new cap changes the amount of the violation, not the reason pooling is wrong.
2. V1 has two true acceptances (A, C), two false acceptances (E, F), one true rejection (B) and one false rejection (D). V3 has three true acceptances, zero false acceptances, three true rejections and zero false rejections. This is a six-case diagnostic, and C still has invalid working.
3. V3 checks an amount, not intermediate steps. Check each claimed allowance against $\min(c_i,h)$ and the total against their sum. C’s first claimed allowance 700 already disagrees with the correct 680.
4. Exact duplicates share their outcome, so four copies are not four independent opportunities; the one-attempt probability cannot be compounded. Even an independently sampled pool containing a success needs a selector that finds it.
5. Keep translations and near-paraphrases in the same family. Freeze untouched families and rule compositions before training, and report student results beside its baseline under the same budget and checker audit. A pass on new numeric values supports value variation, not every new rule or domain.

36.10 References

• Cobbe et al., Training Verifiers to Solve Math Word Problems, 2021, sections 2 and 4. Distinguishes a solution generator from an outcome-trained verifier and runtime selection on GSM8K.
• Lightman et al., Let’s Verify Step by Step, 2023, sections 2–6. Process/outcome reward-model supervision, controlled comparisons, false positives and contamination limitations.
• DeepSeek-AI, DeepSeek-R1, January 2025, sections 2.2–2.4. Rule-based rewards, multi-stage training and SFT distillation as distinct components of one reported recipe.
• Turpin et al., Language Models Don’t Always Say What They Think, 2023, sections 2–4. Counterfactual evidence that visible explanations can omit factors affecting predictions.