Chapter 46 RAG Engineering and Layered Evaluation

46.1 An overall score cannot tell us what to repair

The assistant sometimes refuses a question it should answer. Making the prose friendlier would not reveal whether the source was never retrieved, was discarded while packing, or was ignored by the generator. Conversely, a correct-looking amount can conceal a wrong citation. We need to observe the intermediate states established in Chapter 45.

Chapter 32 required an evaluation contract before accepting a model. KA-4 now has a pipeline contract. The fixed set contains twelve authored diagnostic cases in two languages. It is a development set, including deliberate failures, not a random sample of users. KA-0’s nine-query retrieval comparison and KA-1’s actual model evaluation remain separate, unchanged evidence.

46.2 Give each failure a location and an observable test

Example 46.1 — Four ways to lose the same answer. The required current source is travel-v2. In the retrieval failure, top-one selection returns v1. In the context failure, v2 was retrieved but its evidence is deliberately removed. In the generation failure, full v2 reaches the composer boundary but an authored fault changes the proposed amount to 600. In the citation failure, the proposal keeps 750 but replaces its citation with v1.

Table 46.1 — What to inspect before choosing a repair.

Layer	Evidence to inspect	Local metric or check
Retrieval	Ranked candidates versus required source set	Fraction of required sources retrieved
Context	Exact usable packed evidence, after budget checks	Fraction of required sources actually present
Generation	Proposed answer before the final support gate	Expected task status and amount
Citation	Claimed source, quote, scope and proposed text	Source existence, exact quote, support and claim coverage

The generation check is intentionally narrower than final success. A 750-yuan proposal with a wrong citation can pass amount/status while failing the cited-answer task. The support gate then withholds it. Withholding an unsupported answer is safer behavior, but if sufficient evidence was available and the task required an answer, the end-to-end case still fails. This prevents “always abstain” from scoring perfectly.

Retrieval / context

1. Outside the modelNeed v2
2. An operationRetrieve v1 → retrieval failure
3. Text piecesRetrieve v2, pack none → context failure

A source can be found without reaching usable input.

Generation / citation

1. Outside the modelFull v2 is present
2. Text pieces600 [v2] → wrong amount
3. Text pieces750 [v1] → wrong citation

The final support gate withholds both; the diagnostic labels remain distinct.

Figure 46.1 — Keep the target fixed and locate where evidence breaks. Original teaching diagram · CC BY-SA 4.0

Figure 46.1 keeps the same 750-yuan target while moving the break between layers. The first failing layer is a diagnostic label, not proof that later layers are flawless. We retain all layer indicators, even when the final result happens to pass.

46.3 Change one selection rule and keep everything else

Example 46.2 — Before and after eligibility filtering. Both conditions use the same documents, queries, explicit topic/date inputs, tokenizer, BM25 formula, whole-collection statistics, top-one cutoff, sibling-conflict expansion, full-parent packing, schema, composer and support checker. Only the candidate eligibility filter changes: baseline selects before checking applicability; the improved condition filters by topic/date before selecting. The composer still checks applicability in both.

On “Is 600 still the Beijing lodging limit?”, the baseline selects the obsolete A and then abstains. Filtering removes A for the current date, so B reaches the same composer and produces a supported 750 answer. The historical question still selects A. No model parameters, generator instructions or gold answers changed.

Table 46.2 — All twelve semantic cases, both languages. “Pass” means the final result matches the frozen task contract; a deliberate fault is not excused merely because it was injected.

Case	Required result	Baseline	Filtered
Current lodging	Answer 750 with v2	Pass	Pass
Stale 600 premise	Answer 750 with v2	Retrieval failure	Pass
Historical lodging	Answer 600 with v1	Pass	Pass
Synonym hotel allowance	Answer 750 for the explicit current date	Retrieval failure	Retrieval failure
Passport renewal	Abstain	Pass	Pass
Only expired policy available	Abstain	Pass	Pass
Equal-authority 750/800 conflict	Abstain	Pass	Pass
Insufficient context capacity	Abstain	Pass	Pass
Drop retrieved context	Answer 750	Context failure	Context failure
Change proposed amount to 600	Answer 750	Generation failure	Generation failure
Replace citation with v1	Answer 750 with v2	Citation failure	Citation failure
Cancellation policy	Cite the approval requirement; no action	Pass	Pass

The synonym case is a new RAG task with explicit current date, unlike KA-0’s version-neutral synonym relevance set. Its version mapping is retained; neither set overwrites the other. Fault cases are authored interventions executed by real code, not observations of a model spontaneously making those errors.

Table 46.3 — Actual layer counts; denominators are part of the result.

Measure	Baseline	Filtered	Denominator meaning
End-to-end task success	7/12	8/12	All semantic cases, including expected abstentions
Mean retrieval coverage	8/10	9/10	Ten cases with nonempty required-source sets
Mean usable-context coverage	6/10	7/10	Same ten cases; budget rejection leaves no usable context
Proposed status/amount correct	8/12	9/12	Before final citation gate
Mean supporting-citation fraction	3/5	4/6	Cases whose proposals contain citations
Correct expected abstention	4/4	4/4	Unknown, expired-only, conflict and budget cases

Coverage is calculated per case before averaging. A conflict needs both sources; retrieving one gives $1/2$, not 1. An empty required set is not assigned perfect retrieval. A response with no citation has citation precision marked not applicable, not 1. The proposed-answer citation denominator changes when filtering enables an additional answer, so that row alone is not a matched-case measure of an improved citation checker; the checker did not change. Exact claim coverage is also recorded through the full extractive-text support test. This is not a general natural-language entailment metric.

The context-budget case is correctly refused even though its context coverage is zero. Thus maximizing every layer score independently is not the objective. The end-to-end contract decides success; layer traces explain why.

46.4 More techniques need more specific hypotheses

A filter cannot introduce missing synonym terms. Query rewriting can turn this fixture’s exact phrase “hotel allowance” into “lodging” for retrieval while preserving the original question and explicit scope for generation. The rule is saved, limited and separately tested; it must not silently change “not eligible,” the requested year or the order ID.

Parent-child retrieval separates a compact matching unit from a larger evidence unit. Our authored search card points to a parent; keeping only its amount line loses the scope that expanding the parent restores. Expansion consumes more context, so it needs the same budget accounting.

Compression should be motivated by a measured budget problem. An exact-line compressor can remove a known irrelevant filing note while retaining the annotated support span. A learned summary might lose a negation or date; it needs preservation checks against originals. Hybrid/reranking from Chapter 44 changes candidates and their order, and already has its own same-query comparison. It is not silently added to this filter experiment.

Table 46.4 — Separate actual ablations, not additions to Table 46.3.

Probe	One change	Before → after
English/Chinese synonym	Exact phrase rewrite for retrieval only	Abstain → supported 750, in each language
English current policy	Amount-only span → complete parent	476 → 591 input code points; abstain → answer
English document with an authored irrelevant fourth line	Keep only verified support lines 1–3	1,153 → 591 code points; both answers supported

The filing note is Administrative filing note. repeated twenty times in a separate corpus variant. With its newline and escaped serialization, removing it saves 562 input code points. The support annotation was authored for this fixture, not inferred by a general-purpose compressor. These probes test mechanisms under stated conditions; they are not a new combined winner on a held-out set. A change that helps one row needs a whole-set regression before adoption.

46.5 Repetition and uncertainty need the right unit

Example 46.3 — Twelve cases do not become seventy-two independent cases. Each condition runs twelve cases in two languages, three times: $12\times2\times3=72$ traces, or 144 for both conditions. The local pipeline is deterministic. Repeats check reproducibility; translations share the same scenario. Treating those traces as independent users would understate uncertainty.

Cluster by semantic case. The twelve paired success differences are one $+1$ for the stale premise and eleven zeros. Mean improvement is $1/12=0.083333$, or 8.333333 percentage points. This is a measured improvement on the fixed development cases, not a deployment accuracy estimate.

For a conditional uncertainty calculation, resample twelve cases with replacement from these twelve pairs and average the differences. This paired bootstrap preserves each baseline/improved pair. Here each draw selects the one improved case with probability $1/12$. The number of selected improvements follows the binomial count from twelve such draws. The implementation computes its exact probability distribution by repeated addition of probability masses, not by random model trials. The central percentile interval is [0, 0.25], or [0, 25] percentage points.

The lower endpoint includes no improvement. More fundamentally, resampling hand-designed development cases cannot establish how representative they are of future users. The report makes no significance, model-quality or production-readiness claim. A new evaluation requires representative unseen questions, explicit scope labels, source-version monitoring and adjudication of natural-language support; duplicating current rows is not a substitute.

46.6 KA-4’s completed boundary and the Part VII checkpoint

rag_eval.py saves every intermediate trace and the fixed taxonomy: retrieval, context, generation, citation, or passed. ka4-eval-v1.json specifies expected outcomes, corpus variants, repeats and the single changed condition. data/part-vii/eval-run.json and ablations-run.json retain actual runs. docs/part-vii-validation.md records the executed environment and verification, while docs/part-vii-research.md records external sources. No generation replay is counted as a new neural experiment.

Table 46.5 — Locate the six concepts before handing off.

Concept	Its responsibility here	Evidence of completion
Context	Construct the actual bounded input	Serialized bilingual KA-2 payload and failure branch
Schema	Check object shape and explicit outcomes	Shared schema, typed object and rejection tests
Tool	Request an operation; application actually executes	Ownership-checked read trace and unchanged snapshot
MCP	Exchange descriptions/requests/results consistently	Version-pinned subprocess exchange
Retriever	Rank candidate knowledge	Frozen lexical/dense/hybrid/rerank comparisons
RAG	Use selected evidence to answer or abstain with support	Local extractive pipeline and layered evaluation

The project now connects those interfaces without an accepted general-purpose LLM or a write tool. The original KA-1 gates are still necessary if a model replaces the local composer. The next part asks how a system chooses another step when goals remain open. This chapter hands it bounded operations, explicit state and failure evidence; it does not implement an autonomous decision loop.

46.7 Summary

End-to-end scores judge the task; layered evidence locates failures. A single scope filter repairs one retained case without fixing missing synonyms, corrupted context, wrong generation or wrong citations. Full traces, stable denominators, separate ablations and uncertainty at the case level prevent an appealing answer from substituting for an evaluation.

46.8 Exercises

1. Three relevant sources are required. Retrieval finds two; packing keeps one. Compute retrieval and context coverage. Can a fluent final answer locate which stage needs repair?
2. A system refuses every request in Table 46.2. What end-to-end score does it earn? Why should citation precision on its empty citation set not be called 100%?
3. The one improved case is translated into ten more languages, with identical outcomes. Should the confidence interval shrink merely because the trace count increased? Explain the evaluation unit.
4. You can afford only one next change. Choose between rewriting synonyms, replacing the citation checker and increasing context capacity for the remaining failures. State which row your choice targets and what regression must remain visible.
5. An MCP tool returns the right document; a valid JSON answer cites the wrong paragraph. Identify which contracts have evidence of success, which fail, and who can actually cancel an order in the current project.

46.9 Reference answers

1. Retrieval coverage is $2/3$; context coverage is $1/3$. The trace locates one missing source before packing and another lost during packing. Fluency alone reveals neither. Additional answer and support checks remain necessary.
2. It earns 4/12: only the four expected-abstention cases pass. Citation precision has a zero denominator and is not applicable. Calling it perfect would reward avoiding every answer, which the task contract rejects.
3. No. These are additional views of the same paired scenario, not new independent semantic cases. They may expose language-specific failures and should be reported separately if they do; identical translations do not create new information about case diversity.
4. Rewriting directly targets the synonym retrieval failure; it needs regression on dates, negations, unrelated questions and both languages. Replacing the citation checker would not retrieve missing evidence and might accept bad citations unless independently validated. More capacity does not repair deliberate context removal or wrong amounts; the existing low-budget refusal is already correct. A different observed failure distribution could justify a different choice.
5. Tool delivery and JSON shape can pass. The cited-answer support contract fails at citation selection even if the amount is right. The application owns execution authority, and this project exposes no cancellation tool at all; neither MCP nor valid JSON can supply it.

46.10 References

• Gao et al., Enabling Large Language Models to Generate Text with Citations, §§2–3. Motivates separating correctness, citation support and coverage; our exact-source fixture has a narrower judge.
• Liu et al., Lost in the Middle, §§2 and 5. Shows why retrieved evidence and usable context should be inspected separately under controlled conditions.
• Manning et al., Introduction to Information Retrieval, §8.4. Supports explicit relevance judgments and ranked evaluation; the paired uncertainty calculation above is derived directly for this authored case set.