Chapter 57 Capstone: An Evaluable Personal Knowledge Assistant

Mira is planning a fictional Beijing trip. She needs the lodging limit for the date of travel, the passage that establishes it, and the status of her own order. A fluent answer is useful only if it helps her make the right decision. An assistant that confidently quotes yesterday’s rule, exposes a colleague’s order, or silently repeats a cancellation has failed even when its prose is excellent.

We now have a service that can answer some of those questions. The graduation task is to decide precisely which promise its evidence supports. This chapter follows normal and abnormal requests through the same assistant; it does not start a new chatbot. The foundations are the selection experiment in Chapter 33, the answer contract in Chapter 38, RAG in Chapter 46, recovery in Chapter 49, and the evaluation, governance, security and service work in Chapters 50–56. The HTTP Primer is the practice prerequisite. The default path needs a CPU, not a model account.

57.1 Decide what the assistant is allowed to promise

A requirement is a condition a user can recognize, not a component name. “Use RAG” does not tell Mira whether 600 or 750 yuan applies. Our requirement says: given an explicit date, answer from the eligible fictional policy, show its unit and exact source lines, and abstain when the supported scope is missing. The 2025 rule is 600 yuan per room per night; the replacement beginning January 1, 2026 is 750. Historical questions must continue to work after the update.

The order requirement is separate. An authenticated reader may see an order only when both tenant and owner match the host’s principal. Two tenants may each have an order called A-104; the identifier alone is not authority. The web service offers a read operation. It has no order-cancellation endpoint. The approved cancellation implemented earlier remains a separate local recovery exercise, with its own reviewer, exact action, object version and receipt. Reading an approval policy never grants approval.

Table 57.1 — Requirements, threats and human responsibility. All people, orders and policies here are fictional.

User requirement	What could defeat it?	Acceptance evidence	Human responsibility
The date, amount and unit agree with the cited source	Stale context, missing scope, misleading source text	Current, boundary and historical cases; exact citation checks	A policy owner approves source changes
Only my order is visible	Guessed IDs, a tenant supplied in a question, stolen credentials	Cross-owner and cross-tenant denial; host-owned identity	An access administrator grants and revokes credentials
Stop and failure messages describe what happened	A late result after cancellation; retry after a lost write response	Terminal-state tests, worker-stop trace, receipt recovery	An operator investigates uncertain effects
A faulty candidate can be withdrawn	An unbound evaluation, incompatible data, hidden in-flight work	Candidate digest, rejected gate, switch and rollback journal	A release owner chooses scope and rollback conditions
Feedback improves future tests without exposing users	Self-selected reports, raw private text copied into fixtures	Quarantine, explicit review, deletion and a separate regression	A reviewer decides what the report actually supports

This table is a small threat model: it names valuable assets, ways those assets could be harmed, boundaries that constrain actions, and people who retain authority. It does not assume that all hostile instructions originate in a user’s question. Retrieved documents, tool results and Skill references also carry text. That text can be evidence; it cannot become a new grant of permission. The local process, its operator and reviewed repository files are trusted in this demonstration. A malicious local administrator is outside that boundary. Public demonstration credentials are not secret production credentials.

The scope is therefore deliberately bounded: a local, bilingual assistant for a small dated policy collection and fictional own-order reads. General paraphrase coverage, arbitrary private-document ingestion, public multi-user hosting and autonomous order changes are not accepted requirements. This is a scope decision that must remain visible in the System Card, not an excuse to hide a failing example.

57.2 Follow one answer all the way to its evidence

Example 57.1 — Accepting a 750-yuan answer. Input: the north-tenant demo reader asks “Beijing lodging” with date 2026-09-14, topic travel, and locale en. The fixed collection contains five source cards in the retained A–E order. The two lodging sources have different validity intervals. The question is not merely whether the response contains 750; it is whether each boundary preserves the requirement.

The HTTP handler first turns the demo credential into a host-owned principal. The body cannot choose a tenant. Business control checks the four-field request, the per-user rate budget and the global work limit, then captures a release manifest. A 202 response assigns a request ID; it is not yet an answer. This ID lets Mira’s interface read progress and later obtain the result without submitting a second job.

On a cache miss, the worker reuses the existing RAG implementation. Date eligibility removes the 2025 lodging source from the current-date candidate set. BM25 ranks the remaining eligible cards using the retained recipe; ranking selects evidence, not truth by itself. The context carries the source’s three lines: the scope and unit, the effective date, and the amount. The local answer producer extracts those lines and constructs the answer object. No neural model runs on this default path.

The structured-output check verifies the fields and allowed status. The support check also verifies the source, version, line range, exact quotation, applicable date, complete scope and matching amount. The reviewed source-verification Skill runs its helper and reports support. Only then does the worker publish the final result. The interface renders answer and quotation as text and gives Mira the source version and date. The service records the manifest identity, stages and final outcome for the operator; it does not turn the user interface into an implementation checklist.

Normal request: explicit date 2026-09-14

1. Application recordHost: north / Mira
2. Outside the modelEligible source: travel-v2, lines 1–3
3. An operationStructure + support + Skill checks
4. Application record750 yuan / room / night, with quotation

Default extraction makes zero neural calls. A passing trace explains this answer, not all possible questions.

Abnormal requests and operator decisions

1. Application recordWrong owner → refuse
2. Application recordMissing lexical match → abstain; retain miss
3. An operationStale candidate → reject; local rollback 600 → 750

Keep independent expectations, actual outcomes and human owners together. No average score cancels a permission violation.

Figure 57.1 — An answer trace and a challenged boundary support different acceptance claims. Original book diagram · CC BY-SA 4.0.

Figure 57.1 connects the request to two different kinds of evidence. The upper path explains why a particular answer was accepted. The lower path asks whether the service still behaves acceptably when a boundary is challenged. A successful upper path cannot stand in for the lower one.

In the actual HTTP record, this input returns answered, 750 and travel-v2. The Chinese counterpart does too. Changing only the date to 2025 returns 600 from travel-v1. Asking about a volcano returns abstained with no quotation. These are measured local executions in data/part-ix/service-run.json, not measurements of a language model’s general knowledge. The complete requests, events and results remain available; the textbook keeps the decisive facts directly visible.

A fourth request exposes a limit: “overnight accommodation ceiling” also returns abstained. The corpus contains a relevant policy, but the narrow lexical path does not recover it from that wording. Abstaining avoids inventing a number, yet still fails the user’s supported-information need. We retain this example instead of adding its synonym to the dictionary and silently replacing the measurement. A proposed rewrite rule needs its own version, paired cases and evidence that it does not change a historical question into a current one.

57.3 Keep comparisons honest when the system grows

The project began with retrieval, so we retain its original comparison. In KA-0, every method receives the same nine queries, five cards, candidate order and success rule. A query counts as successful when its first selected source is acceptable, including the specified empty result for an out-of-scope question. This is a source-selection task; it is not the service’s answer-and-permission task.

Table 57.2 — Fresh rerun of the unchanged lexical comparison. Counts are successful queries out of nine in each language.

Method	English	Chinese	What changes relative to the previous row?
Keyword overlap	5/9	5/9	Count distinct matching vocabulary items
BoW Count with cosine	6/9	6/9	Preserve repetition and normalize vector length
TF-IDF with cosine	7/9	7/9	Also reduce the influence of common terms

For example, 7 successes and 2 failures give $7/9\approx77.8\%$. The denominator is nine queries, not five documents and not the number of words. Both languages use the corresponding fixed content resources. The actual rerun preserves 5, 6 and 7 successes. Their narrow same-set comparison remains useful even though the later system uses date eligibility and BM25. We cannot call a ten-case release gate an improvement from 7/9 to 10/10: the tasks and denominators changed.

Model selection also retains its unfavorable result. Chapter 33’s two trained Mini GPT candidates did not satisfy the declared bilingual KA-1 answer contract. Chapter 38 separated formatting adaptation from factual correctness. Their saved runs and checkpoint provenance remain in the evidence pack; Part IX does not retrain them. The selected service uses an extractive producer because its small answer scope can be checked directly. This choice sacrifices fluent general synthesis and broad paraphrase coverage. It is not evidence that extraction is best for every personal assistant.

The optional Qwen3 model runs from Chapters 47 and 50 remain historical evidence for a bounded decision adapter and a judge experiment. In the latter, an authored travel calibration matched 8/8 presentations, while the separate public human-judgment comparison agreed on 0/4 cases. Repeated presentations of four travel cases are not eight independent human judgments. Neither result justifies turning that judge into the graduation authority. The default release gate uses independently authored expected status, amount and source; it makes zero new neural calls.

Example 57.2 — Why one graduation score would mislead. Suppose a report adds the TF-IDF successes, the good candidate’s release successes and the blocked attack cases: $7+10+6=23$. It divides by $9+10+6=25$ and announces $92\%$ quality. The arithmetic is correct, but the quantity has no agreed meaning. A successful source choice, a dated answer and a rejected hostile proposal are different events. The attack cases were deliberately constructed, and the release cases overlap the same tiny policy collection. Adding copies of easy cases would increase the number without improving Mira’s experience.

Keep these denominators separate. Chapter 50’s two injected variants each pass 3/4 end-to-end tasks, yet one first fails retrieval and the other first fails answer production. The fresh capstone rerun preserves those different first failures. Diagnosis determines the change to try; the final percentage does not. A permission violation is a stop condition even if thousands of easy policy answers pass.

57.4 Challenge the authority and recovery boundaries

Example 57.3 — The same order ID is not the same order. North’s Mira reads A-104 and receives processing. South’s Leo reads A-104 and receives shipped. These are actual local results from distinct fictional tenant records, not contradictory model answers. The host combines the principal with the object ID before reading the store. Mira’s attempt to read someone else’s order is refused. Attempting to add tenant: "south" to the four-field HTTP request is rejected as an invalid body.

The service does not cache order reads. Its policy-response cache includes tenant, user, scopes, the full request and release digest; authorization runs again before use. In the CPU regression, another tenant cannot read a known job ID either. Hiding an order button would not establish any of these properties: the checks must hold for direct HTTP requests.

The capstone also reruns the earlier read-only MCP exchange through an actual subprocess. It discovers the one read tool, reads the owned A-104 snapshot, denies B-205, and rejects cancel_order as an unknown tool. That teaching transport uses the earlier KA-3 snapshot. The HTTP service uses the later tenant-aware KA-6 store; the two snapshots must not be described as one live external order system. The protocol subset, version and exchanges are recorded. Tool annotations describe intent; host enforcement establishes authority.

Six fixed attack cases then compare the deliberately unsafe fixture path with guarded handling. They challenge a false current amount, document-driven exfiltration, cancellation hidden in a tool result, a tenant argument in a Skill reference, a prohibited fictional secret and active HTML output. All six compromise the unsafe local fixture; none compromises the guarded path in this run. No external request is sent. This evidence checks the named controls against these proposals. It does not estimate success against adaptive attackers or certify an arbitrary model agent.

Example 57.4 — A lost response does not erase a cancellation. The separate approved-write exercise commits one fictional cancellation, then injects a lost response. The request enters UNCERTAIN; the order already has one effect. Reopening the SQLite store and resuming the same approved intent reuses its receipt. The final effect count remains 1, and the request becomes DONE. Reading the stored trace adds 0 mutations. Compensation remains manual_review, with 0 automatic business writes.

There are three different observations here: uncertainty in the caller, committed state in the business store, and a replayed record. None is a second authorization. A fresh idempotency key could describe a new intent and defeat duplicate suppression; inventing one after a timeout is not recovery. The HTTP demonstration avoids this write path entirely, while preserving its existing recovery evidence as part of the project.

57.5 Make the acceptance decision reviewable

Table 57.3 — The capstone evidence ledger. “Fresh” means executed for this increment; “retained” identifies earlier measured evidence whose files are checked by hash.

Requirement or claim	Evidence and observed result	Decision and remaining boundary
Reproduce a baseline	Fresh same-set 5/9, 6/9, 7/9 per language	Keep the original corpus and misses
Select an answer producer	Retained failed Mini GPT selection; fresh source-checked extraction	Accept the narrow producer; no general model-quality claim
Return dated, structured, cited answers	Fresh current/historical/boundary and missing-evidence cases	Accept those cases; synonym miss remains open
Enforce read authority and recover approved writes	Fresh 20-row permission matrix, read-only MCP exchange, one-effect receipt recovery	Accept tested boundaries; web writes remain absent
Resist the fixed attack proposals	Fresh 6 guarded cases; unsafe comparisons retained alongside them	Accept this regression; adaptive attack coverage remains limited
Make the service usable	Actual HTTP cancellation/errors and bilingual keyboard/browser checks	Local demonstration; no full assistive-technology conformance claim
Control release and failure	Gates 10/10, 10/10, 4/10; stale candidate rejected; 600 → 750 rollback; breaker recovery	Local configuration drill; no executable/data-migration rollback claim
Govern reports and observations	Quarantined category, maintainer-reviewed regression, deletion; explicit SLO window	Small in-memory monitor; no production availability estimate

The local candidate can be demonstrated under the scope in Table 57.1 because its required boundaries have executable evidence and its failures have owners. The candidate that forces the old date fails the gate and is rejected. The rollback drill deliberately bypasses that gate only inside a controlled local exercise, then records the incorrect 600 answer, the restored 750 answer and the old in-flight request that still returns 600. A rollback decision must account for work already accepted under the faulty snapshot.

The fault drill provides a second warning against superficial graduation. Two good outcomes among five eligible jobs give 40% in that deliberately damaged window. A successful half-open probe restores future calls; it does not erase the three bad outcomes. This confirms that the monitor and recovery path expose the injected incident. It is not a claim that 40% is an acceptable user-facing SLO, or that a short later success proves production readiness.

We also keep failures found while building the increment. A real cancellation was initially counted as a downstream failure because Python’s InterruptedError was caught by a broader OSError handler. A behavior assertion exposed the misclassification; the corrected trace stops the worker without that failure event. In the first service browser run, four checks failed because the interface expected order_id while the store supplied id. The service boundary now projects the store field into its public result, and all six service browser checks passed on rerun. These are repair records with a causal code change and a repeatable failing condition, not conclusions inferred from a dashboard trend.

57.6 Hand over an operable project, with limits intact

The evidence pack is organized around the shared assistant. code/knowledge-assistant/ contains the implementation; it is not copied once per part. code/part-ix/README.md gives the clean CPU setup, test and notebook commands. data/part-ix/ holds independent expectations, actual request/fault/release records, candidate manifests and execution hashes. The two executed notebooks let a practice reader reconstruct service and operations observations without a paid API. Their printed outputs are evidence of execution, not merely cells that can be opened.

The English handover in docs/part-ix-capstone.md maps requirements to those artifacts and records the evaluation decision. The accompanying runbook explains startup, health inspection, fault diagnosis, switching and rollback. The Data Card records fictional provenance, immutable old versions, permitted use and deletion scope. The System Card records the extractive default, host authority, resource limits and rejected claims. The Risk Register names an owner, mitigation, evidence and unresolved condition for each risk. The license and dependency inventory separates original CC BY-SA 4.0 content/data/figures, Apache-2.0 code and third-party components. A hash identifies bytes; it does not grant their license or prove they are safe.

An operator should be able to answer a concrete incident question with this pack: which request failed, which manifest it captured, which first boundary failed, whether an effect committed, which requests remain in flight, and which evidence supports recovery. If the answer requires the original author to remember an undocumented step, the handover is incomplete. A candidate file alone cannot restore changed executable bytes or reverse a database migration; this local drill changes compatible configuration and explicitly records that limit.

Some responsibilities remain human. The policy owner must approve a future source version. An access administrator must manage real identity before any real-user service is considered. A maintainer must review feedback and regression changes. A release owner must authorize a wider exposure only after new load, security and accessibility evidence. The current standard-library HTTP server, public demo credentials, bounded in-memory job store and single-machine tests do not supply those missing capabilities. They are sufficient for the stated local exercise, not a hidden industrial deployment template.

The independent multimodal elective after this chapter is optional. It is not a missing graduation gate, and adding it would not repair the synonym failure, a permissions defect or an untested rollback. The next useful project step follows the evidence: choose one unmet requirement, preserve its baseline, propose a bounded change, and measure that change against both the new case and the old obligations.

57.7 Summary

A completed assistant is a set of justified promises. We traced a dated answer from identity through evidence and validation, challenged authority with abnormal requests, and separated stored replay from recovery of a committed effect. The unchanged lexical baseline, rejected model candidates, live service observations and local release/fault drills remain distinguishable. Graduation accepts the demonstrated local scope while retaining misses, risks and named human decisions; it does not convert a collection of passing checks into general reliability.

57.8 Exercises

1. A proposed 2027 lodging source changes the limit to 800 yuan. Specify the historical, boundary and current cases, cache/version changes and source approval needed before accepting it. Explain why editing travel-v2 in place is a poor comparison.
2. A report combines 18 successful retrievals out of 20 with 100 blocked repetitions of one attack and advertises 118/120 quality. Calculate that fraction, then identify two reasons it cannot replace a release decision.
3. After an approved cancellation’s response is lost, the interface displays “Please try again.” Explain what information the host must preserve, what the user can safely be told, and why generating a new intent key is dangerous.
4. The same-set synonym regression improves after a rewrite rule, but a historical question now receives 750. Should the candidate pass? State the requirement that decides, the comparison to preserve, and one focused next experiment.
5. A maintainer points to six browser checks, ten release cases and one successful recovery probe as permission to host this service publicly. Identify three missing kinds of evidence or capability. Distinguish a missing capability from an observed local test failure.

57.9 Reference answers

1. Keep a case before the new effective date, a case exactly on it, and a later case; also retain the existing 2025 and 2026 queries. If the new interval begins January 1, 2027, December 31, 2026 must still return 750, while January 1, 2027 and a later date return 800 with the new source, unit and lines. A policy owner approves the fictional rule and its interval. Add a versioned source, update eligibility/index and release hashes, and bind cache entries to that version; do not let old responses answer the new interval. Editing the old source would destroy the baseline and could make past answers unverifiable. Run permission, support and failure regressions as well as the new cases.
2. $118/120\approx98.3\%$. Retrieval correctness and blocked attack proposals have different units of evaluation, so their sum has no declared user meaning. Repeating one attack 100 times does not add 100 independent threat conditions and lets that easy case dominate the number. Report 18/20 retrieval separately and describe the single attack condition with repetition count. A permission violation can still block release regardless of the aggregate.
3. Preserve the authenticated principal, exact approved action, object version, original intent/idempotency key, expiry, attempt/deadline budget and durable receipt lookup. Tell the user that completion is being checked; a missing response does not establish failure. Reconcile the original intent against the receipt before deciding whether any retry is allowed. A new key may look like a new authorized intent and permit an additional effect. Reading the trace alone changes nothing and cannot settle a missing business receipt.
4. No. The dated-answer requirement is a hard gate; a synonym gain cannot pay for a wrong historical amount. Retain the old rule, exact paired inputs, expected sources and both outcomes. Test whether the rewrite preserves the explicit date and topic before retrieval, using current and historical paraphrases that differ in only that condition. A candidate that restores date fidelity can then be compared on the same old and new cases.
5. Examples include real credential issuance/revocation and tenant administration; a suitable hardened serving stack and independent access review; representative sustained-load and recovery tests with durable telemetry; and assistive-technology/user testing beyond the current keyboard and browser checks. These are missing capabilities or untested contexts, not evidence that the recorded local cases failed. The observed synonym miss is an actual functional limitation. A successful probe establishes one recovered call; it neither supplies these capabilities nor estimates long-run availability.

Part IX concept checkpoint. Explain the service to a colleague using one normal and one abnormal request. Identify who owns identity, why 202 is not completion, when a citation may be shown, what cancellation acknowledges, what binds an evaluation to a release, what rollback leaves in flight, why a cache hit needs scope, what belongs in the SLO denominator, and who may promote feedback into a regression. Finally name one promise this project has not earned.

Part IX checkpoint answer. The host authenticates and authorizes each request; 202 assigns work before it finishes. The default worker extracts eligible evidence and verifies structure and support before the interface shows the final quotation. A successful cancellation acknowledgment requests stopping; the terminal cancelled record and stopped worker establish the observed outcome. A manifest digest binds candidate bytes/configuration to its gate. Rollback redirects new requests while old snapshots may continue. Cache keys preserve identity, date, request and release scope after authorization. The declared window counts eligible good and bad outcomes under an explicit latency rule; it cannot silently exclude overload or erase an incident. Feedback stays quarantined until a maintainer reviews a separate, minimized regression. Public production readiness and broad paraphrase coverage remain unearned; the retained synonym miss is a concrete reason for the latter.

57.10 References

• NIST, Artificial Intelligence Risk Management Framework: Generative AI Profile, NIST AI 600-1 (2024). MEASURE 2.3–2.6 and MANAGE 4.1–4.3 support contextual evaluation, documented limitations, feedback and incident responsibilities. The project is not a NIST certification.
• Google, The Site Reliability Workbook: Canarying Releases (2018). Explains limited exposure, comparison and rollback considerations. Our compatible local configuration drill does not demonstrate distributed deployment.
• Google, The Site Reliability Workbook: Implementing SLOs (2018). Supports explicit user-relevant indicators, windows and error-budget policy; the injected smoke window is not a production reliability estimate.
• OWASP, LLM09:2025 Misinformation. Supports source verification and appropriate human review. Visible citations alone do not establish correctness.
• W3C, Web Content Accessibility Guidelines 2.2. Provides the accessibility criteria underlying the label, focus and status checks. Passing this small browser suite is not full conformance evidence.