Chapter 50 From Model Evaluation to System Evaluation

Two versions of the assistant pass three of four requests. Should we make the same improvement to both? In the first version, the current policy never reaches the answer stage. In the second, the historical policy arrives correctly but the proposed amount is wrong. The totals agree; the repairs do not.

Chapter 32 separated an evaluation contract from a model’s apparent fluency. Chapter 46 saved intermediate RAG evidence. Chapters 47–49 added choices, tools, budgets and recovery. A system evaluation must now ask both whether the user got an acceptable result and whether the route to that result respected the contract.

50.1 Keep the target fixed while locating the failure

A model can choose a valid action name and still choose it for the wrong reason. Retrieval can return a relevant document without its unit reaching context. A correct final answer can accompany an unauthorized cancellation. These are different observations, so a single answer score cannot represent them all.

Table 50.1 — What each layer needs as evidence.

Layer	Input and observation to preserve	Check in this project
Model / bounded choice	Exact decision input, model digest, output and stop reason	At most one call; allowed value; decision quality measured separately
Retrieval	Question, date, candidate order, scores and selected sources	Applicable source found, or correct empty result
Context	Complete serialized input and source spans	Evidence survived packing with its scope and unit
Generation / citation	Proposed answer, parsed fields and supporting quotations	Structure, amount, exact span and source eligibility checked separately
Tool / trajectory	Host principal, proposal, approval reference, budget and receipt	Authorized action; finite attempts; no duplicate effect on recovery
End to end	Required user outcome and all preceding safety conditions	A supported answer, appropriate abstention/refusal, or verified action result

A layer marked “not reached” must not be silently counted as a pass. It may also be inappropriate to count it as an independent generation error when retrieval supplied nothing. Keep the first observable failure and the later consequences. The earliest observed defect is a useful repair lead, not proof that no other defect exists.

The KA-1 Mini GPT acceptance decision remains unchanged. The optional Qwen choice adapter is a separate component with separate records. Likewise, the original nine-query lexical comparison is retained: Keyword passes 5/9, BoW Count 6/9 and TF-IDF 7/9 in each language. Adding a sophisticated workflow does not justify dropping a cheap baseline or changing its adverse cases. Those nine queries and the four cases below have different contracts and cannot share one denominator.

50.2 Equal totals can hide different systems

Example 50.1 — Two systems at 3/4. Use four fixed requests, in this order: current lodging, historical lodging, cancellation-approval policy, and a question with no supported topic. The first requires 750 on 2026-09-14; the second requires 600 on 2025-09-14. The third requires the cited approval rule, not cancellation of an order. The fourth requires abstention. These are diagnostic teaching cases, not a random sample of users.

System R deliberately loses the current-policy corpus at retrieval. System G retains retrieval but deliberately changes the historical candidate amount to 750. Other processing is the same. The final support gate withholds the bad candidate rather than publishing it. We execute both variants on the same ordered cases.

Table 50.2 — Every outcome behind the equal totals.

Request	Required outcome	System R	System G
Current lodging	Answer 750 with v2	Fails: no source, abstains	Passes
Historical lodging	Answer 600 with v1	Passes	Fails: proposes 750; gate withholds
Approval policy	Cited policy answer	Passes	Passes
Unsupported topic	Abstain	Passes	Passes

R passes the last three rows; G passes the first, third and fourth. Each has three successes divided by four required request outcomes, so $3/4=0.75$. The unsupported-topic row belongs in the denominator because correctly declining is its required outcome. A number such as “answer accuracy among answers actually returned” would use a different denominator and could conceal R’s missing answer.

In R’s first row, retrieval fails and the downstream positive-answer checks are not applicable. In G’s second row, retrieval and context pass, the structure accepts an integer amount, but the amount does not match the historical requirement or source. That gives generation/support evidence for the failure. A schema check alone cannot detect this: both 600 and 750 are integers.

For a fixed request $i$, write $e_i=1$ if its complete acceptance contract is met and $e_i=0$ otherwise. The end-to-end score for $n$ requests is:

$$\text{success rate}=\frac{e_1+e_2+\cdots+e_n}{n}.$$

Here $n$ counts requests under this contract, not tokens, returned citations or executed tool calls. For R the numerator is $0+1+1+1=3$; for G it is $1+0+1+1=3$. The calculation loses the positions of the failures, which is precisely why we retain the rows.

Evidence path

1. Outside the modelTarget: current = 750
2. An operationRetrieve v2 → retain lines 1–3
3. An operationChoose → compose → check

Save each intermediate artifact before assigning a failure layer.

Equal final score, different repair

1. Application recordSystem R: retrieval misses one case
2. Application recordSystem G: historical answer says 750
3. Text piecesBoth succeed on 3 of 4

Fix candidate selection in R; inspect generation/support in G. A total cannot choose the repair.

Figure 50.1 — Follow a failure upstream. Original teaching diagram · CC BY-SA 4.0

Figure 50.1 connects the final result to intermediate artifacts. Fixing R’s evidence selection and fixing G’s answer production are different interventions. If both were wrong in one request, changing only retrieval could reveal a previously hidden generation failure. Attribution therefore requires rerunning the affected path, not declaring victory when the first error disappears.

50.3 Test software, evaluate models, monitor service use

A software test can inject cancel_order into the two-value decision slot and check that no tool call occurs. It can reopen a database after a lost reply and count effects. These tests are repeatable because the expected behavior follows from explicit rules. They do not measure how often a real model proposes an invalid value.

A model evaluation records prompts, outputs, decoding settings and failures across a fixed set. Repeating a stochastic run can expose variation, but ten repetitions of one request are not ten independent kinds of user need. Temperature zero also does not turn a tiny set into a coverage guarantee. Report malformed output and truncation instead of discarding those records before calculating a score.

Online monitoring sees actual service use: failure rates, latency, refusal frequency and reported problems. It also sees selection bias. If only 20 of 100 users provide feedback and 17 are positive, $17/20=85\%$ describes the responding group. It is not automatically an 85% satisfaction estimate for all 100. Users who leave, never reach the feedback screen or dislike the answer may be missing for systematic reasons. Stratified review and deliberately sampled follow-up can reveal what voluntary feedback misses; they still need a stated collection policy.

These three activities complement each other. A unit test proves a narrow behavior under its inputs, an evaluation estimates performance under its sample and settings, and monitoring detects changes in operation. None can certify the other two simply by passing.

50.4 A Judge needs an independent reference

An LLM Judge is a model asked to assess another answer. It is attractive when exact matching is too narrow, but its verdict is another model output. The rubric, reference evidence and parsing rules become part of the evaluator, and therefore need evaluation too.

First use the authored travel contract: current 750, old 600, cancellation not approved. The four answers are “current 750,” “current 600,” “current 750; old 600,” and “current 750; order cancelled.” The first and third pass; the second and fourth fail because their claims are unsupported. The local Qwen Judge returned those four verdicts in both languages: 8/8 matches to the authored oracle. After those outputs were fixed, one real reader independently supplied the same four labels under the stated evidence rule and consented to anonymous calibration use. All four unique cases agree in each language. That is eight model decisions against four human-labeled meanings, not eight independent human annotations. The separate human-label file records provenance, consent and limitations; a second annotator was not obtained.

We also test transfer using four previously human-labeled first-turn pairs from the research authors’ public MT-Bench dataset, licensed CC BY 4.0. The extraction rule was fixed before local Judge execution: source order, distinct question IDs from 100 upward, first turn only, and a small input-length limit. It selects an English convenience slice. One retained human vote per pair is a reference judgment, not a unanimous or infallible truth.

Example 50.2 — A human calibration that rejects the Judge. In the race question, answer A says that overtaking the second runner puts you first and that runner second. Answer B says you are second and that runner is now third. Track the order: the runner who was first has not been passed, so B follows the stated move. The retained human label prefers B. The local Judge preferred A in the original order and returned a tie after swapping the candidates. Its easy travel results did not transfer to this elementary case.

Table 50.3 — All four human-reference cases and measured decisions. “Swapped” has already been mapped back to the original A/B identities.

Case and decisive comparison	Human preference	Original order	Swapped, mapped back	Conservative result
Imagine being a threatened tree: A follows the creative perspective; B declines because it is an AI	A	B	B	B
Overtake the second runner: A says first/second; B says second/third	B	A	Tie	Tie
White House riddle: A invents a position among colored houses; B names Washington, D.C.	B	Tie	Tie	Tie
Healthy Thomas visits a hospital: A mostly suggests treatment; B also includes work and caregiving	B	A	A	A

The conservative rule keeps a preference only when it survives the order swap; otherwise it returns a tie. None of the four aggregated decisions matches its retained human label: agreement is $0/4$. Three pairs are consistent across positions, so consistency is $3/4$. The tree and hospital decisions are consistently different from the human vote; the riddle is consistently a tie. Consistency is therefore not correctness.

The resulting decision is to reject this configuration as an automatic release Judge. This is not a claim that it always performs badly, nor a reproduction of MT-Bench’s published aggregate results. The set is tiny, selected for manageable reading, English-only and drawn from a different task distribution than travel QA. The successful single-reader travel calibration adds narrow domain evidence; it does not cancel these failures or establish agreement on harder travel questions. Full original pairs, source revision, row positions, human labels and raw local outputs remain in the fixed data.

50.5 Check what a high score might be rewarding

Position, length and self-preference require controlled comparisons. The underlying question and content should remain fixed where possible; otherwise a quality difference can masquerade as bias.

Table 50.4 — The additional local probes, including their limits.

Change	Actual observation	What follows?
Swap supported 750 and unsupported 600	Judge prefers 750 in both orders	This particular clear pair survives the swap
Repeat the same correct sentence three times	Judge prefers the short answer in both orders	No preference for repetition in this one pair; not a universal absence of length bias
Keep answers identical, swap “this judge model” / “another model” labels	Judge selects the first position in both orders	A source-label experiment is confounded with position; it does not establish self-preference
Generate four new Qwen answers, compare each with the original human-preferred answer, swap order	The Judge prefers its own answer in both orders for 2/4 pairs	Actual self-generated text was tested, but these new pairs have no independent human quality labels

The last experiment is deliberately separated from human calibration. The old human vote concerns the original two answers; it cannot be reassigned to a newly generated Qwen answer. For example, Qwen’s new race answer correctly says “second” but also says the passed runner is ahead. That mixed result shows why a name or a single correct phrase cannot establish equal quality. The 2/4 own-answer preference is an observation, not proof of a causal self-enhancement bias.

Zheng and colleagues discuss position, verbosity and possible self-enhancement effects, and explicitly retain uncertainty about the latter in their study. Our response is to preserve swaps, quality references and disagreements, rather than treating “use an LLM Judge” as a complete evaluation method.

50.6 Count the errors that matter to a decision

Example 50.3 — A score can conceal unsafe acceptance. This is a separate hand-counted exercise, not the observed Qwen confusion matrix. Ten answers have independent reference labels: five should pass and five should fail. A hypothetical Judge accepts three of the five good answers and one of the five bad answers. It rejects the remaining two good answers and four bad answers.

Table 50.5 — Count reference quality before naming the metrics.

Reference label	Judge accepts	Judge rejects	Row total
Should pass	3	2	5
Should fail	1	4	5
Column total	4	6	10

Seven decisions agree with the reference: three correct acceptances plus four correct rejections, giving accuracy $7/10=70\%$. Among the four accepted answers, only three should pass, so acceptance precision is $3/4=75\%$. Among the five answers that should pass, the Judge accepts three, so recall is $3/5=60\%$. The denominators differ because each question selects a different group of cases.

If this Judge automatically releases answers, the one accepted bad answer is especially consequential. If it sends every rejection to a reviewer, the two rejected good answers create review work. Choosing a threshold therefore requires costs, capacity and a risk tolerance, not just a larger overall percentage. For the actual local configuration, the human-reference disagreement is already a reason to keep automatic release disabled.

50.7 Practice and the KA-7 regression report

system_eval.py produces the same-score matrix and its intermediate traces. The model probes are saved separately from deterministic fault runs. judge-calibration.json maps swapped labels, retains ties, calculates the authored-oracle, consented travel-human and public human-reference results separately, and records the rejection decision. The Notebook recomputes these quantities from the saved raw records without contacting a model service.

The regression report also checks the actual tool invariants from Chapter 49: denied approvals produce zero effects, reopened recovery produces one, and replay makes zero mutations. Model-choice, retrieval, context, generation/citation, tool/trajectory and end-to-end observations are all represented, but they are not averaged into an arbitrary universal score. The failure-attribution report identifies a repair owner and a rerun target for each failure class.

50.8 Summary

Evaluate the system’s route as well as its final answer. Keep request-level denominators, intermediate artifacts, negative cases and cheap baselines. A Judge can agree with a human on four simple domain cases yet disagree with human judgments on other tasks. Swapping candidates can expose inconsistency but cannot make a wrong rubric or weak evaluator correct. KA-7 retains both the successful software controls and the rejected automatic Judge, making its acceptance boundary explicit.

Quality evidence still does not settle privacy, fairness, rights or responsibility. The next chapter follows those obligations through the data lifecycle.

50.9 Exercises

1. System R is changed to omit abstentions from its score. It now reports three correct answers out of three returned answers. Is the current-policy failure repaired? State both useful denominators.
2. A trace contains v2 after retrieval, an empty context, and an abstention. Which layer should be investigated first? Would a generation-temperature change address the observed defect?
3. In Table 50.5, a stricter Judge rejects the previously accepted bad answer but also rejects one more good answer. Recalculate accuracy, precision and recall; explain the tradeoff.
4. Four identical-answer pairs are always judged “A wins.” They are perfectly consistent under repeated runs. What should an order-swap test reveal, and why is repeated-run consistency insufficient?
5. A new self-preference study replaces answer B with a newly generated answer but keeps the old human label. Diagnose the mistake and specify a valid comparison.

50.10 Reference answers

1. No. Answer precision among the three returned answers is 3/3, but request-level success remains 3/4 because one required answer is absent. Both can be useful if named honestly; neither can silently replace the other.
2. Inspect context selection or packing first: the evidence was found and then lost. Changing temperature does not restore missing input. After repairing packing, rerun generation and support checks because a later defect may still exist.
3. The counts become two correct acceptances, zero bad acceptances, five correct rejections and three rejected good answers. Accuracy remains $(2+5)/10=70\%$; precision is $2/2=100\%$; recall is $2/5=40\%$. Accepted answers are cleaner in this fixture, but more useful answers are withheld. If nothing were accepted, precision would be undefined rather than automatically perfect.
4. After swapping, “A” names the other underlying answer. Mapping labels back reveals that the preferred identity changes. Deterministically choosing a position is repeatable behavior, not evidence of quality judgment; identical supported answers should tie under the stated rubric.
5. The human label refers to a particular pair, not merely the question. Preserve the original pair for existing-label calibration, and obtain new independent quality labels for the new pair. Hide author identity and swap positions; record ties and disagreement instead of treating an own-answer win as proof of self-bias.

50.11 References

• Liang et al., Holistic Evaluation of Language Models, §§2.1–2.4 and §4. The scenario/adaptation/metric separation supports explicit evaluation contracts; our application layer matrix is original.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, §§3.3–3.4 and 4.1. These sections describe bias probes, position swapping and human comparison, including the limits of self-enhancement claims.
• LMSYS MT-Bench human judgments. The pinned four-case adaptation uses the human split under CC BY 4.0, retaining the first-turn text and labels; it is not the paper’s full benchmark.