Chapter 38 Diagnosing Capability Gaps: Prompt, RAG, Tools, or Parameter Updates

38.1 Before choosing a method, locate the failure

KA-1 was asked Old: 600. New: 750. Old limit: . Its selected checkpoint replied 750.. Both amounts had reached the model, and the question explicitly asked for history. Fetching the same facts again would not by itself explain how to repair this response.

On a different request, a real order status might never have reached the system at all. That is another failure. A model can describe what “processing” means without accessing an order database. These cases need different evidence before we choose an intervention.

Chapters 32 and 33 established the KA-1 contract and rejected both local checkpoints as bilingual assistants. Chapter 22 separated training stages from capabilities. We will now keep the failed cases, make a small actual runtime change, and ask which larger changes the evidence supports or rules out. The methods are options at different layers, not successive levels of a universal upgrade ladder.

38.2 Separate an absent source from an unused source

A knowledge freshness gap occurs when the system lacks the relevant current source. A behavior gap occurs when the needed evidence is present but the answer does not use it as requested. A format gap concerns the representation required by the consumer. A domain or language gap is a suspected mismatch between training coverage and the target distribution. A task capability gap concerns operations the system cannot reliably perform, even with relevant inputs. A single failure can involve several gaps; these are diagnostic hypotheses, not labels that identify a cause automatically.

Table 38.1 — What the existing failures establish.

Evidence	Supported diagnosis	What it does not establish
English current answer 750. conveys the supplied amount but fails literal 750 matching	A format mismatch on this response	A missing fact or universally poor instruction following
Historical prompt supplies old=600 and new=750; answer is 750.	Requested scope was not used correctly	That retrieval omitted 600
Chinese prompts produce unrelated English fragments	Failure on the bilingual target distribution	That continued pretraining is the unique remedy
Supplied fictional status produces blue.	Failure to use this given status	A failed live database query; none occurred
No approval is granted, yet the answer is France.	The required approval boundary was not answered	That the system actually cancelled anything
Passport information is explicitly unknown; answer is new.	Missing required abstention	That a passport rule can be derived by longer reasoning

A useful counterfactual changes one condition. Supply a known correct source while keeping the question and evaluation fixed. If the answer improves, investigate source delivery; if it still fails, inspect serialization, scope and the model’s ability to use evidence. Then vary current versus historical scope without changing the supplied records. Such a comparison is more informative than assigning every wrong answer to “insufficient knowledge.”

KA-1 already supplies compact evidence; it is not an implemented retrieval-augmented generation system. Its status for fictional A104 is an added authored assignment. The older KA-0 FAQ describes a possible processing status; it did not query an account. The original A-104 case and KA-1’s shorter fictional label remain separate records.

38.3 Repair one narrow thing and keep the judge unchanged

Example 38.1 — An actual paired runtime adaptation. The frozen extension makes one change: on the three amount tasks only, if a valid UTF-8 response consists entirely of ASCII digits followed by one terminal period, remove that period. Surrounding whitespace may be removed for this match. Return the same digits. Every other decoded response is preserved.

Thus 750. becomes 750, while 750 ye is t., not 750, 750.0 and status blue. remain unchanged. The rule does not know the correct amount. It never substitutes 600 for 750, repairs a sentence by extracting its first number, or changes an approval response. It is a conservative output-format normalization at runtime, not SFT, RAG, a new sampling strategy or a learned capability.

The experiment retains all six bilingual cases, both MG-2 checkpoints, greedy and temperature-0.7/top-5 policies, seeds 7, 42 and 31415, the 24-new-token cap, rubric and quality gates. It actually makes 144 local generations. Each generation supplies the raw and normalized display conditions, so paired responses share exactly the same sampled output. The three greedy repeats measure repeatability and timing; they are not independent questions.

We selected this repair after inspecting the retained KA-1 failures. These are development results, not a blind acceptance test.

Table 38.2 — All selected-checkpoint primary responses. Each row’s output repeated identically on all three greedy runs. The Chinese-column strings are actual outputs, not translations. Leading whitespace is retained in the data.

Request	English raw → normalized	Chinese raw → normalized	Semantic score, English / Chinese
Current amount	750. → 750	yuor: blue. → unchanged	2 / 0
Correct stale premise	750. → 750	France. → unchanged	2 / 0
Historical amount	750. → 750	France. → unchanged	0 / 0
Supplied order status	blue. → unchanged	France key France. → unchanged	0 / 0
Ungranted approval	France. → unchanged	e key France. → unchanged	0 / 0
Unknown passport rule	new. → unchanged	. → unchanged	0 / 0

A semantic score of 2 means correct content, scope and boundary; only 2 passes. The original AI-assisted author judgments are reused only after every raw output reproduces the retained run exactly. Removing a terminal period from an integer amount preserves the factual claim, including the wrong historical claim. The report exposes that reasoning; no new human panel or hidden judge API is claimed.

38.4 A format gain that does not pass the assistant gate

Table 38.3 — Paired greedy results under the original criteria. There are 18 attempts and six distinct cases per model/language. A robust semantic case passes only if all three attempts score 2. Every arrow below means raw → normalized.

Checkpoint and language	Strict passes	Semantic passes	Robust cases	Decision
Selected step 50, English	0/18 → 6/18	6/18 → 6/18	2/6 → 2/6	Reject
Selected step 50, Chinese	0/18 → 0/18	0/18 → 0/18	0/6 → 0/6	Reject
Final step 600, English	0/18 → 0/18	0/18 → 0/18	0/6 → 0/6	Reject
Final step 600, Chinese	0/18 → 0/18	0/18 → 0/18	0/6 → 0/6	Reject

For selected English, nine displays change: three tasks times three repeats. Only six become strict successes because the historical amount stays wrong. A claimed “33.3 percentage point accuracy gain” would conceal that the gain is exclusively literal format matching. Semantic quality and the original requirement of six robust cases in each language remain unchanged.

The stress policy is retained separately. Selected English changes three displays: the current case at seed 42, and the historical case at seeds 42 and 31415. Only the current answer is correct. Strict passes move from 0/18 to 1/18; semantic passes remain 1/18 and robust cases remain 0/6. All other stress groups still have zero semantic passes. The final checkpoint’s invalid UTF-8 response is retained, not “repaired” by dropping replacement characters.

Table 38.4 — Actual cost of the same primary generations. Times are milliseconds, rounded to six decimals. The normalized time is the measured generation interval plus the immediately following normalization interval. Empirical p95 uses the nearest-rank rule. Token counts include EOS and are identical in the paired conditions.

Checkpoint and language	Raw p95, ms	Normalized p95, ms	Generated tokens
Selected step 50, English	0.982250	0.982542	66
Selected step 50, Chinese	2.185542	2.186292	114
Final step 600, English	5.020875	5.021208	207
Final step 600, Chinese	5.186250	5.186958	156

The run used Python 3.12.10, PyTorch 2.7.0, one CPU thread and float64 on arm64 macOS 26.5.2. Models were warmed; loading was excluded. Full-prefix recomputation, trace collection and byte decoding were included in generation. These tiny-model times are not service latency promises. Normalization saves no generated tokens: the period has already been generated. Checks of the model tensors confirmed no parameter changed. A displayed string becoming shorter is not evidence of reduced model computation.

The extension was frozen at 2026-09-13T12:23:33.409374+00:00 and the run recorded at 2026-09-13T12:23:36.621417+00:00. Inputs and implementation hashes are in ka1-format-freeze-v1.json; complete responses, probabilities, bytes, stops, costs and comparisons are in data/knowledge-assistant/ka1-format-run-v1.json.

Current amount

1. Outside the modelRequired: 750
2. Text piecesRaw: 750.
3. An operationRemove terminal period → 750
4. Outside the modelStrict pass; meaning was already correct

Three repeats on current + three on stale premise = six new strict passes.

Historical amount

1. Outside the modelRequired: 600
2. Text piecesRaw: 750.
3. An operationRemove terminal period → 750
4. Outside the modelStill wrong; semantic gate unchanged

Three further displays change; their factual claims do not. Both checkpoints remain rejected.

Figure 38.1 — The same format repair preserves both the correct amount and the wrong one. Original diagram of the recorded KA-1 comparison · CC BY-SA 4.0

Figure 38.1 follows the two amount questions through the same repair. Their strict-match outcomes differ because their required meanings differ. This is why we keep the evaluator fixed while changing the system, rather than quietly broadening the judge and calling that an adaptation.

38.5 Match the intervention to the missing layer

The following choices are recorded in the continuing project’s docs/ka1-adaptation-report.md. They are decisions and proposed next tests, not already implemented future chapter interfaces.

Table 38.5 — Support, cost and veto for each option.

Option and changed location	Evidence supporting a trial	Main cost	Veto or limit for KA-1
Prompt: input wording, ordering and examples	Supplied old/new scope is mishandled; inspect template and paired scope prompts first	More input tokens and a development comparison	Cannot supply an unavailable fact or grant permission; MG-2’s 64-token context leaves little room
RAG: retrieve versioned external evidence into input	KA-0 has synonym and stale-premise retrieval failures; changing policy needs a current source	Index maintenance, retrieval evaluation and added context	Current KA-1 already receives the amounts; retrieval alone cannot certify correct use
Tool: call a program or an external data service	A live status needs an actual lookup; exact calculations can use an explicit rule	Interface, result validation, failures and access control	No live tool exists here; a status FAQ is not a query, and querying does not authorize cancellation
Continued pretraining: adapt parameters to domain text	The target language/domain differs from the small training distribution	Curated corpus, training resources and retention evaluation	Chinese failures alone do not isolate this cause; no evidence justifies this cost for the teaching model
SFT or preference training: adapt behavior through demonstrations/comparisons	Repeated evidence-use, abstention and boundary failures motivate representative labeled data	Data review, training and regression testing	A few corrected KA-1 answers risk memorization; labels cannot substitute for current sources or permissions
Runtime formatting or extra computation	A demonstrated format mismatch; a candidate pool may contain better answers	Wrapper/check costs or additional generation	The actual repair changes format only; more computation needs useful alternatives and a capable selector

RAG stands for retrieval-augmented generation: retrieving evidence and making it available as input to a generator. This chapter chooses where it could help; later chapters build its interfaces. A tool is a callable program or service with a specified operation and result. Here it may be a calculator or order lookup. A cancellation operation would additionally require its own authorization and safeguards; producing text about it does not execute it.

These methods can be combined. Retrieval can supply the latest policy, a calculator can apply a declared reimbursement rule, and instruction training can improve how the model uses both. Every added component also creates a new place to fail. Before combining methods, record which failure each is expected to fix, what evidence would falsify that expectation, and what original cases must remain correct.

Table 38.6 — Retained source-selection baselines, separate from answer quality.

Same nine KA-0 queries	English	Chinese
Keyword	5/9	5/9
BoW Count	6/9	6/9
TF-IDF	7/9	7/9

The new run recomputed these unchanged comparisons. TF-IDF still misses the synonym request and accepts the archived source for the stale-premise request. Its 7/9 is source-selection success, not an answer-generation score. We do not pool it with KA-1’s six-case results.

38.6 A decision can be to stop upgrading

The completed comparison supports retaining the normalizer as a narrow teaching diagnostic. It does not support deploying either checkpoint as a bilingual assistant or beginning expensive parameter training on six public questions. The next justified investigation is a paired input/template and scope audit, with a new development version if inputs change. Source freshness and real order access require separately evaluated external capabilities later.

The MG-2 training mix is tiny and predominantly English. This makes language coverage a plausible hypothesis, but architecture, capacity, optimization and prompting are confounded. “Chinese failed, therefore continued pretraining is required” would skip the evidence. Likewise, “more training steps must help” already conflicts with the retained checkpoint comparison.

If a later adaptation is tuned on these failures, keep this suite as a regression set and create genuinely unused acceptance families before choosing the next model. Preserve historical, unknown and approval cases rather than remove them to improve a percentage. Compare each language and failure category under the same criteria and budget. A useful adaptation report includes rejected options and remaining failures, not just its preferred method.

Only after evidence supports a parameter change do we need to choose its form. That is Chapter 39’s question: what do continued pretraining, SFT, full updates and low-rank updates each specify?

38.7 Summary

Diagnose a failed output by tracing source delivery, requested behavior, format, distribution coverage and required operations. The actual KA-1 repair raises selected English strict matching from 0/18 to 6/18 while leaving robust semantic performance at 2/6; both checkpoints remain rejected. Prompt, RAG, tools and parameter updates change different layers and can be combined when evidence justifies each. An upgrade without a retained baseline cannot establish what it improved.

38.8 Exercises

1. Nine selected-English responses change but only six become strict successes. Reconstruct both counts and explain why historical errors remain.
2. A new evaluation removes the historical and approval cases before reporting improvement. Is this comparable to KA-1? What should the report preserve?
3. A current-policy document is successfully retrieved, but the model repeats the archived amount. Choose the next diagnostic. Then consider a case where the document never reaches the input.
4. A tool returns the correct status, but the assistant says the order has been cancelled. Which two boundaries need checking? Would LoRA alone establish them?
5. A proposed domain training job reports only lower training loss and no Chinese retention results. State what is missing before recommending it.

38.9 Reference answers

1. Three amount tasks each have three repeats: $3\times3=9$ changed displays. Current and stale-premise tasks correctly require 750, giving $2\times3=6$ strict successes. The historical task requires 600, and the rule preserves every digit of the wrong 750.
2. No. It changes the task denominator and removes known failures. Preserve all six original cases and report a separately named subset only as an additional view. A new task contract cannot inherit the old acceptance claim silently.
3. With evidence present, inspect template/IDs, truncation and paired current/history prompts, then whether the model can use the evidence. With evidence absent, inspect retrieval, version selection and context delivery. Repeating the same retrieval is not a demonstrated fix for ignoring it.
4. Check faithful use of the tool result and the distinction between reading status and permission to cancel. An actual cancellation needs an authorized execution path; a parameter-update method does not create or verify that path.
5. Missing evidence includes a stable target-distribution split, baseline and adapted results under the same rubric, both languages, historical/unknown/approval failures, retained capabilities, data rights and declared resource costs. Training loss measures the training objective, not assistant acceptance.

38.10 References

• Gururangan et al., Don’t Stop Pretraining, 2020, sections 3–4. Domain/task-adaptive pretraining and relevant-domain controls in RoBERTa classification experiments; supports a hypothesis to test, not a KA-1 remedy already demonstrated.
• Ouyang et al., Training language models to follow instructions with human feedback, 2022, sections 3 and 5. Behavioral training and evaluation tradeoffs; does not supply evidence of current policy access or tool authorization.
• Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, 2024, sections 3–5. Motivates checking task difficulty, verifier quality and budget before allocating extra runtime work.