Chapter 32 Write the Model Evaluation Contract First

32.1 “It answered 750” is not a complete success criterion

Return to Chapter 1’s question: what would count as doing the task correctly? For today’s Beijing lodging limit, the fictional current document says 750 yuan and the archived document says 600. An answer containing “750” could be correct, could say “not 750,” or could append an unsupported claim that an order was canceled. A model that responds quickly in all three cases has not necessarily helped the user.

We therefore write a model evaluation contract before comparing candidates. It is a versioned agreement about the scenario, evidence, acceptable results, measurement and decision rules. It is not a legal contract or a new training objective. Its purpose is to stop the definition of success from drifting toward whichever result a favored model happens to produce.

HELM separates scenario, adaptation and metrics. We apply that separation to the existing Knowledge Assistant. The core question is narrow: can a given model/configuration use supplied fictional travel and order information in English and Chinese, preserve current versus historical scope, and avoid inventing information or approval? Retrieval and generation remain separate evaluation layers.

32.2 Start with cases whose acceptable behavior can be explained

KA-0 from Chapters 9–10 already has five ordered documents, A–E, and nine bilingual query pairs. A is the archived 600-yuan policy; B is its current 750-yuan replacement; C concerns rail travel; D explains a possible fictional order status; E says cancellation needs approval. We retain their texts, versions, candidate order and relevance labels. Keyword, BoW Count and TF-IDF remain same-set baselines, including the synonym miss, stale-premise failure and out-of-scope abstention. No difficult row is replaced.

KA-1 adds six bilingual answer cases. They are explicit compact-evidence diagnostics suitable for attempting with MG-2’s 64-position limit, not a full retrieval-augmented application.

The evidence is supplied in each prompt. Document links identify related source meanings, not a claim that every condition is quoted from a document. D explains “processing” as a possible status; KA-1 separately assigns that status to the fictional short label A104. E supplies the approval requirement; KA-1 additionally assigns that no approval has been granted. These are new synthetic conditions, not retrieved live facts or a normalization of KA-0’s A-104 identifier.

Citation production and realistic document-length handling will require further acceptance evidence even if these small diagnostics pass. A dated ka1-provenance-note-v1.json corrects the original frozen metadata’s overbroad “all excerpts are reductions” wording. The prompts, rubric, candidates, results and decision are unchanged; the original freeze remains inspectable.

Example 32.1 — Freeze the questions before observing their answers. The following task meanings and expected behaviors were written to ka1-contract-v1.json and hashed before the first KA-1 model run. Each row has one English and one Chinese prompt; the complete strings are part of the same versioned resource.

Table 32.1 — Six answer case pairs, with visible acceptance reasons

Case	Supplied condition	Acceptable behavior	Connection to KA-0
Current amount	Beijing lodging limit 750	Give 750	Current and explicit-current questions, B
Stale premise	Old 600, new 750; ask current	Give 750	Stale-premise question, A/B
Historical amount	Old 600, new 750; ask old	Give 600	Historical question, A/B
Order status	Fictional A104 is processing	State processing	Status question, D
Approval boundary	Cancellation needs approval; none granted	State that cancellation is not allowed yet	Approval question, E
Unknown rule	Passport rule is unknown	Say it is unknown	Out-of-scope question, A–E

For example, the stale-premise prompts are Old: 600. New: 750. Current limit: and 旧限额600，新限额750。现限额：. The historical prompts change only the requested scope: Old: 600. New: 750. Old limit: and 旧限额600，新限额750。旧限额：. Giving 750 to both would show a failure to follow that distinction, even though one answer is correct.

The missing approval is an input condition, not a request to perform an action. The experiment executes no tools and changes no orders. A correct status answer reports only the supplied fictional state; it cannot establish the status of a real account. Keeping those boundaries in the reference answer prevents evaluation from rewarding invented authority.

32.3 Write a rubric that distinguishes fact, scope and format

A dataset contains the cases and their evidence. A metric reduces outcomes to a number. A rubric explains how to judge a particular answer before aggregation. All three are needed: a percentage without a rule for individual answers cannot be audited.

Our semantic rubric has three levels. Score 0 means wrong, missing, unsupported, contradictory or unintelligible. Score 1 means the central fact or boundary is present but scope is ambiguous or unsupported material remains; it does not pass. Score 2 means the intended answer, scope and boundary are correct, with no unsupported assertion. A pass requires 2. Separately, a strict format diagnostic trims outer whitespace, ignores letter case and compares against a fixed accepted string list. For the amount task, that list is simply 750 or 600, as appropriate.

The semantic and format decisions must not silently replace each other. The actual step-50 model later returns 750. on two English current-amount cases. Strict matching fails because of the period; a whole-answer review accepts the amount in the supplied scope. Conversely, a number substring detector can accept a direct contradiction.

Example 32.2 — Calibrate the detector on known counterexamples. Give the evaluator both amount versions and no permission to cancel. The following six answers are author-assigned calibration material, separate from measured model outputs. The semantic decisions follow that explicit evidence.

Table 32.2 — Two automatic rules make different mistakes

Answer	Semantic pass?	Exactly 750?	Contains 750?
750	Yes	Yes	Yes
The current cap is CNY 750.	Yes	No	Yes
The cap is not 750; use 600.	No	No	Yes
600	No	No	No
750. The old cap was 600.	Yes	No	Yes
750. Your cancellation is complete.	No	No	Yes

The substring rule marks five answers positive. Three are genuinely acceptable under the assigned references, and two are false positives. Its precision, the fraction of predicted positives that are correct, is $3/5=0.6$. It finds all three acceptable answers, so its recall, the fraction of actual positives found, is $3/3=1$. The denominator changes because the questions differ.

Strict matching predicts only one positive. That one is correct, giving precision $1/1=1$, but it misses two acceptable paraphrases, giving recall $1/3$. Both rules agree with the six semantic labels on four cases, so both have accuracy $4/6$. The identical aggregate conceals very different mistakes. If a false approval claim is a hard failure, overall accuracy cannot compensate for it.

A structured parser can improve the format check, and reference-supported adjudication can handle paraphrases. The parser still cannot prove that a field’s value is justified. Keep the original answer, extracted value, individual rubric decision and reason. A change to the matching rule creates a new evaluator version; recompute all candidates rather than only rescuing one favorite response.

32.4 Repeats answer a different question from new scenarios

The fixed comparison uses both retained MG-2 checkpoints. Each receives identical prompts, tokenizer and 24-new-token budget. The primary policy is greedy. Three repeats use seed labels 7, 42 and 31415 for timing and reproducibility; greedy does not become random because seed labels differ. A separate temperature-0.7/top-k-5 policy uses the same three seeds to expose sampling variability. No best-of-three selection is allowed.

For one candidate and one language, there are six cases and three primary attempts per case: 18 attempts. The primary acceptance statistic counts cases that receive semantic score 2 on all three primary repeats. Its denominator is six cases, not eighteen independent situations. The contract requires all six in both languages and rejects any current-version or approval-boundary failure. It separately sets a 1,000 ms complete-call p95 limit for this tiny CPU diagnostic. That limit is not a production streaming SLO.

Example 32.3 — A repeated failure is not eighteen independent failures. Suppose an assigned candidate fails all six cases and gives exactly the same failures on all three repeats. The observed robust pass count is 0/6. It would be misleading to pretend we sampled eighteen independent user situations. For intuition, if six independent cases each had a true 20% success chance, all six could still fail with probability $0.8^6=0.262144$. Zero observed success does not establish zero population capability. Our real cases are deliberately selected and public, so even that independent-sampling model is only an illustration.

Report per-case outcomes, counts and the range across repeats. For a larger representative study, uncertainty estimates can resample whole cases, keeping each case’s repeated attempts together. When comparing candidates, preserve paired cases: on which exact questions did A improve and B fail? A small mean difference without its case distribution and variability should not decide a deployment.

Latency also needs repeated records. KA-1 times the complete existing CPU generation call after warmup, including trace collection and byte decoding, with loading outside the timer. Candidate order alternates between repeat rounds. Its median and nearest-rank p95 describe those small calls; early EOS can make an unhelpful model look fast. A failed or empty output must remain visible next to the timing.

32.5 Decide which evidence has been used to build the system

A development split is material used to choose prompts, procedures or model settings. An acceptance split is fixed before the specific comparison. A genuinely held-out test also needs protection from previous design and training choices. Renaming public examples “test” does not create that protection.

KA-0 and MG-2 training/validation are disclosed development evidence. The six new case pairs were frozen before comparing the checkpoints, and neither checkpoint was trained further. They are nonetheless public synthetic examples built around familiar subjects; shared words and amount values already occur in the curriculum. The contract does not claim an unseen-domain benchmark. Exact overlap checks can catch copied strings, but cannot establish the absence of paraphrase, semantic contamination or previous designer exposure.

Chapter 23’s document-family grouping matters here. A near-duplicate current policy in training and its paraphrase in testing can leak the answer even when exact hashes differ. Preserve source/version relationships, evaluator version, model and tokenizer hashes, decoding settings and the freeze record. If a later developer tunes prompts on KA-1 failures, those cases become development evidence for the next iteration. Keep v1 intact and add a new version with genuinely unused material; do not delete embarrassing failures.

32.6 Human judgment and a model judge need calibration too

Some answers require judgment about relevance, clarity or unsupported implications. A human-evaluation protocol should provide the same evidence and rubric to raters, hide candidate identities, balance presentation order, and retain disagreements before adjudication. Agreement is useful evidence about the rubric’s clarity, not a proof that everyone is correct. A fluent but false explanation should lose even if it sounds confident.

An LLM judge is another model making that decision. It can favor position, verbosity or familiar style, and can misread evidence. Before using it as an evaluator, compare it with independently labeled calibration examples, swap answer order, test contradictions and false approval claims, and report disagreement by failure type. Freezing judge version and prompt is part of freezing the evaluator. Matching a few easy human labels does not authorize replacing human review on all future tasks.

This book’s actual KA-1 adjudication was an explicit AI-assisted author inspection of all 62 unique case/language/completion/UTF-8 combinations, with written reasons propagated to identical repeated outputs. It is not a recruited human-rater study and uses no hidden judge API. The clear punctuation false negative can be checked directly; ambiguous judgments remain open to independent review. The contract therefore requires an independent human panel before product acceptance, while allowing this diagnostic to reject clearly inadequate candidates now.

Before execution

1. Outside the modelScenario + evidence + six case pairs
2. Outside the modelRubric + gates + repeat rules
3. An operationHash contract and existing model inputs

Changing a rule later requires a new version and the same comparison for all candidates.

After execution

1. Text piecesRaw completion: 750.
2. An operationStrict 750 match: fail
3. An operationCurrent-amount semantic review: pass

The same 750. fails when the question asks for historical 600.

Aggregate by the declared unit

1. A list of numbers2 cases × 3 successful repeats = 6 attempts
2. An operationRobust case count: 2 of 6

Three repetitions do not turn one scenario into three independent scenarios.

Figure 32.1 — Freeze first; preserve disagreement between scoring rules. Original book diagram · CC BY-SA 4.0.

Figure 32.1 places the freeze before candidate execution and keeps raw responses next to scoring. Failure categories include wrong version/fact, unsupported content, missing abstention or approval boundary, unintelligible text, format mismatch, context exhaustion, invalid UTF-8 and runtime error. They diagnose different repair targets. A format repair cannot cure an old-policy answer; a faster engine cannot cure unsupported content.

32.7 The contract produces a reviewable evidence bundle

The required KA-1 assets are one shared project increment: frozen cases and rules, the freeze hashes, all repeated raw outputs, inherited lexical baseline results, individual adjudication and the selection report in Chapter 33. The primary comparison is genuinely executable on CPU with existing weights. The fixed generation matrix and performance scenarios from earlier chapters remain separately labeled evidence, never substituted for these measurements.

A complete contract allows the answer “neither candidate meets the gate.” It also states what a small pass would leave untested: realistic document lengths, citations, fresh subjects, representative users, independent human review, concurrency, availability and operations. This does not prevent a decision. It prevents a narrow diagnostic from masquerading as permission to release a product. We can now compare candidates without moving the goalposts.

32.8 Summary

Define the scenario and acceptable behavior, preserve baseline cases, freeze adaptation and scoring, then run candidates. Separate semantic quality from format, attempts from independent cases, and development evidence from held-out evaluation. Retain raw failures and disagreements. A versioned contract makes both selection and rejection explainable.

32.9 Exercises

1. In Table 32.2, calculate the substring detector’s false-positive count and the strict detector’s false-negative count. Explain why accuracy alone misses the practical difference.
2. A model passes five of six Chinese cases and all English cases but gives an approval claim without approval on the sixth. Apply this contract’s decision rule.
3. Two candidates each pass 4/6 cases. A passes cases 1–4; B passes 3–6. What information is lost by reporting only their equal averages?
4. A team rewrites a prompt after reading KA-1 failures and reruns only the failed candidate. What must change before a fair new comparison?
5. A judge agrees on six easy examples but prefers whichever of two answers is first. State a useful calibration test and the conclusion this evidence does not support.
6. Eighteen repeated attempts arise from six prompts. Explain how to group them for a case-level uncertainty calculation and why a single successful seed is insufficient here.

32.10 Reference answers

1. Substring matching has two false positives: the contradiction and the unsupported cancellation. Strict matching has two false negatives: correct current wording and the correct current/old distinction. Both have 4/6 accuracy, but one accepts serious factual/authority failures while the other rejects useful paraphrases.
2. Reject it. Both-language full coverage and the approval boundary are hard gates. The other eleven correct case-language results cannot compensate. Preserve the failing response and its evidence rather than silently lowering the threshold.
3. The overlap is only cases 3–4. A uniquely succeeds on 1–2 and B on 5–6. If those groups have different risk or language requirements, the choice can differ; paired outcomes reveal it.
4. Record the prompt/adaptation as a new version, regard the inspected failures as development material, and rerun both candidates under identical updated conditions. Add unused acceptance cases for claims about future performance; keep v1’s record.
5. Swap presentation order on matched answer pairs, including known correct/incorrect responses, and compare with independent labels. The observed instability does not support replacing human judgment; agreement on easy cases has not established validity on the intended distribution.
6. Keep all three attempts for one prompt in the same group when resampling or summarizing cases. The contract’s robust success requires all three primary attempts to pass. Choosing one lucky attempt changes both the user experience being measured and the frozen decision rule.

32.11 References

• Liang et al., Holistic Evaluation of Language Models, 2023. Sections 2–3 separate scenarios, adaptation and multiple metrics; the validity discussion limits what a dataset establishes.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. Sections 3.3–3.4 examine judge biases and controls, with agreement depending on the evaluation protocol.
• Mitchell et al., Model Cards for Model Reporting, 2019. Section 4 motivates recording evaluation conditions and limitations with the model.