Chapter 33 Choosing an API or Local Model from Evidence

33.1 A selection report is allowed to select neither

Chapter 32 fixed the Knowledge Assistant’s success criteria before running candidates. We can now answer a concrete question: do either of the two available CPU configurations meet that contract? The answer is no. One configuration handles two English current-amount cases better, and both run quickly on these tiny inputs, but neither meets the bilingual fact, history, status and approval requirements.

That is a completed selection decision, not a missing winner. A report that must name a winner regardless of the gates would reward the least inadequate candidate as though it were adequate. KA-1 instead preserves the evidence, identifies what each configuration can support, and states what would have to change before another comparison.

The report uses vendor-neutral labels A and B for the local comparison. A is MG-2’s validation-selected step-50 checkpoint; B is the same architecture at step 600. These are genuinely different trained parameter configurations, not fabricated large-model benchmarks. A lower validation loss previously selected A, while B had lower training loss. Neither loss substitutes for the application contract.

33.2 Identify exactly what was compared

Example 33.1 — The measured comparison record. The run was recorded on 2026-09-13 under ka1-contract-v1, whose cases and input hashes were frozen before execution. Both candidates use the existing 13,875-parameter Mini GPT, context length 64, the same byte-BPE tokenizer, float64 CPU arithmetic, and one thread. No checkpoint was retrained or converted. Parameter tensors contain 111,000 bytes per model: $13{,}875\times8$. This excludes Python, PyTorch, activations and file-container overhead; it is not process peak memory.

The environment was Python 3.12.10, PyTorch 2.7.0 and macOS 26.5.2 on arm64, installed in a fresh temporary CPU environment using the repository’s pinned dependencies. The primary greedy policy and the temperature-0.7/top-k-5 stress policy each ran three seeds on six cases in each language. There are $2\times2\times3\times6\times2=144$ attempts: candidates, policies, repeats, cases, languages. All attempts remain in the record.

Loading was outside timing; each candidate received two warmup calls. Timing covers the whole existing generation function, including trace collection and byte decoding. It recomputes the full prefix and has no KV cache or network. The latency measurements therefore cannot establish production TTFT, TPOT, concurrency or server availability. A very short wrong answer may complete especially quickly.

Table 33.1 — Measured primary results; three repeats of each case

Candidate and input language	Strict passes / 18 attempts	Semantic passes / 18 attempts	Cases passing all repeats / 6	Median / p95 complete-call ms
A, English	0	6	2	0.718 / 0.983
A, Chinese	0	0	0	1.376 / 2.230
B, English	0	0	0	1.515 / 5.476
B, Chinese	0	0	0	1.023 / 5.160

The p95 uses the nearest-rank convention over eighteen calls per row; durations are rounded to three decimals here and retained at greater precision in the raw record. Eighteen timings do not establish a reliable production tail. A’s six semantic passes are the same two cases repeated, not six distinct successes. All primary outputs are valid UTF-8; their poor Chinese performance is a content problem, not merely byte decoding.

33.3 Inspect the responses that produced the aggregate

The prompts are the fixed compact evidence in Chapter 32. All three greedy repeats produce the same continuation for each candidate/case/language. The following two tables therefore display all distinct primary responses without hiding a favorable or unfavorable seed. Model text is reproduced unchanged, including broken words; it is not edited into fluent English or Chinese.

Table 33.2 — Measured English continuations

Requested answer	A	B
Current 750	750.	750 ye is t.
Correct stale premise to 750	750.	750ped.
Historical 600	750.	750 ye the blue.
Fictional status: processing	blue.	t: te. Answer: shine. Answer: te.
No approval, so not allowed	France.	bje. Order bje. Ae. Us: red
Unknown passport rule	new.	new.

Table 33.3 — Measured Chinese-prompt continuations

Requested answer	A	B
Current 750	yuor: blue.	red red red t: red te. Ver: red t: red t
Correct stale premise to 750	France.	.
Historical 600	France.	.
Fictional status: processing	France key France.	suor.
No approval, so not allowed	e key France.	er: red the red the red the red er:
Unknown passport rule	.	.

A’s 750. is a whole-answer punctuation false negative for the strict detector on the first two English cases. On the history question, exactly the same string is wrong. B’s correct-looking numeric prefixes do not make its complete fragments intelligible answers. The review reasons in ka1-adjudication-v1.json distinguish these situations; a substring detector would blur them.

B’s English status and Chinese current-amount outputs hit the 24-new-token budget. Its approval outputs in both languages hit the context limit. Other primary outputs stop at EOS. Stopping is a mechanism, not a success label: an EOS after new. does not express “unknown.” The approval failures do not imply that an order was actually canceled; the experiment has no action execution at all.

The stress policy gives A one semantic pass among eighteen English attempts, with no case passing all three; all other candidate/language rows have zero semantic passes. One B English stale-premise attempt produces invalid UTF-8, displayed as 750�on: n: �. while the original bytes are retained. It is a recorded decoding failure, not a reason to delete the attempt. No strict-format stress attempt passes. This variability reinforces rejection and supplies failure evidence; it is not eighteen independent estimates of population quality.

The inherited retrieval comparison also ran again on all nine KA-0 queries in both languages: Keyword 5/9, BoW Count 6/9, TF-IDF 7/9, matching the retained baseline. Those numbers measure document selection. They are not added to the answer successes to create a larger “assistant accuracy.” Keeping them separate points to different repair targets.

33.4 Compare hard constraints before weighted preferences

Table 33.4 — Evidence and remaining obligations for the local candidates

Dimension	A and B evidence	Decision consequence
Ability and language	Table 33.1–33.3; A only 2/6 English robust cases, both 0/6 Chinese	Both fail the quality gate
Latency	Small warm CPU calls below the diagnostic 1,000 ms p95 gate	Speed does not repair incorrect answers; no service SLO proven
Cost	No paid inference calls; tensor storage counted	Hardware, electricity, idle capacity and maintenance cost unmeasured
Privacy	Local runner loads repository data/weights and makes no service requests	Prompt egress avoided in this experiment; deployed logging/access still needs review
License	Original book data/checkpoints CC BY-SA 4.0; original code Apache-2.0	Preserve per-asset terms; do not label the whole package with one license
Availability	Runs with retained local files in the tested CPU environment	No API quota dependency; no uptime or multi-user resilience test
Maintenance	Small inspectable model, fixed tokenizer, explicit tests and hashes	Owners still need dependency updates, data versions, recovery and monitoring

Some dimensions are gates. A requirement to keep a particular dataset local can eliminate an external route before quality testing. An unmet current-policy or approval rule eliminates these local candidates despite low latency. Weighted preferences are useful only among candidates that satisfy the hard constraints. Otherwise, cheapness can mathematically “compensate” for behavior the application explicitly forbids.

If two feasible candidates trade latency against operating effort, record why the intended workload values one more. If A is no worse on every measured preference and better on at least one, it dominates B on those measurements; missing privacy or language evidence still cannot be assumed favorable. A leaderboard’s task mix rarely equals the application’s full constraint set.

Measured candidate A

1. Stored model numbersStep 50; 13,875 parameters
2. A list of numbersEnglish 2/6; Chinese 0/6
3. An operationQuality gate failed → reject assistant use

750. is correct for current scope and wrong for historical scope.

Measured candidate B

1. Stored model numbersStep 600; same architecture
2. A list of numbersEnglish 0/6; Chinese 0/6
3. An operationQuality gate failed → reject assistant use

Low training loss and short calls do not establish acceptable answers.

Evidence retained for a new comparison

1. Outside the modelContract + hashes + 144 raw attempts
2. Outside the modelReview reasons + unchanged KA0 baselines
3. An operationNew candidate → same or explicitly versioned contract

API and larger local candidates have dated documentation only; no substituted benchmark.

Figure 33.1 — Hard quality gates decide before cost preferences. Original book diagram · CC BY-SA 4.0.

Figure 33.1 ends the local comparison with two rejections and a preserved evidence bundle. A remains a useful validation-selected teaching checkpoint, and B remains useful for studying memorization and overfitting. Neither is accepted as the bilingual Knowledge Assistant. A retrieval baseline that remains useful does not transform either generator into a released product.

33.5 Price the workload rather than the model name

Example 33.2 — An assigned monthly cost comparison. Suppose two hypothetical candidates have already passed the same quality and privacy gates. The API costs USD 2 per million billed input tokens and USD 10 per million billed output tokens. Each request is assigned 1,000 billed input and 200 billed output tokens, with no cache, tools, retries or discounts. Its cost is $1000\times2/10^6+200\times10/10^6=0.004$ dollars per request.

A hypothetical local service costs 20 dollars per month in fixed capacity plus 0.001 dollars per request in variable cost. At 1,000 requests, API cost is 4 dollars and local cost is 21. At 10,000, API cost is 40 and local cost is 30. Let R be monthly request count. Equality requires

$$0.004R=20+0.001R,\qquad R=20/0.003\approx6{,}666.67.$$

The left side is API dollars; the right side is local fixed plus variable dollars. For whole requests, the local estimate becomes cheaper from 6,667 per month under these assumptions. It does not establish that the local machine can serve that traffic at the required latency, or that staff time is included. Failed attempts and retries belong in cost per successful request. Different tokenizers also produce different billable counts for the same text; do not copy Mini GPT’s token count into another provider’s bill.

This example is deliberately vendor-neutral. Actual quoted prices, product versions and license terms belong in a dated verification layer, separate from the stable decision method.

33.6 Dated online verification: documentation is not a benchmark

The companion external-candidates-2026-09-13.json records as_of: 2026-09-13 and last_verified: 2026-09-13. It examines one hosted and one downloadable candidate through official sources. Neither was executed under KA-1, and neither can be chosen on the basis of the CPU results above. They illustrate what to verify before an optional future experiment.

Table 33.5 — Official documentation inspected on that date

Candidate	Verified identity and stated conditions	Unmeasured or unresolved here
Hosted API example	Claude Haiku 4.5, snapshot claude-haiku-4-5-20251001; listed base rates USD 1/MTok input and 5/MTok output; model page lists 200K context and 64K maximum output	KA-1 quality in either language, request latency, account eligibility and actual bill
Downloadable example	Qwen2.5-0.5B-Instruct; card lists 0.49B total parameters, 24 layers, Q14/KV2 and 32,768 context; inspected Apache-2.0 license text	No download/run; local speed, memory peak, task quality, complete stack and model revision pin

The API identity and rates come from its model specifications and official pricing table. At those base rates, an assigned 1,000-input/200-output request would cost 0.002 dollars before other charges. This is an estimate, not an invoice. Listed access is subject to the supported-region policy; mainland China is absent from the inspected commercial API list. The example is not a proposal to bypass access conditions.

The provider’s commercial terms and organization-retention policy answer different questions from price. A no-training statement does not mean zero storage. The inspected standard API retention rule has a 30-day timeframe with service/agreement, enforcement and legal exceptions, and longer rules for some flagged or feedback data. A particular organization must verify its applicable contract and data path; this book does not approve submission of private records.

For the local example, read the specific model card, config and actual license. Its family introduction mentions longer contexts, but this model’s own limit is 32,768. The config’s last-change revision begins eaa56b5; the license was introduced at f0ef055. The record retains full file-change hashes and does not misrepresent them as a pin of the entire current repository. Before a real run, pin the complete weight/tokenizer/config revision and tested engine as well.

Open weight means weights are made available under some terms; the term alone says nothing complete about freedoms, training code or data information. The OSI Open Source AI Definition 1.0 identifies additional requirements, including the preferred form for modification. An Apache-licensed weight release is useful license evidence, not automatic proof that every system component satisfies that definition. Conversely, local execution can avoid an external prompt service while still leaking through application logs or integrations if those are designed poorly. Architecture, access, licensing and operations are separate axes.

33.7 KA-1 decision and the next question

The completed report rejects A and B for the stated assistant use, retains A for the existing teaching role, and preserves both checkpoints and every failure for follow-up. Reconsideration requires a new frozen comparison with an appropriate candidate, unchanged or explicitly versioned task criteria, full language coverage and new evidence for any previously missing requirement. No API/GPU experiment, independent human-panel validation, production load test or deployment is claimed.

The evidence package contains the contract, pre-run freeze, model/tokenizer/data/code hashes, 144 raw attempts, adjudication reasons, lexical same-set comparison and this dated decision. The accompanying English engineering report provides commands and machine-readable file paths. Core reading requires neither running code nor purchasing access; the visible tables suffice to audit why these candidates fail.

At the end of this part, follow one request across all layers. Data provenance and mixture affect what training can learn; parameters, tokens and compute constrain the run; components and architecture shape its state; generation chooses outputs; inference and serving consume resources; evaluation determines whether the result is acceptable. Improving one link does not certify the rest. The next question is therefore motivated by evidence: why does a base model trained to continue text still need separate teaching to follow instructions and assist a user? Chapter 34 begins there.

33.8 Summary

Choose against a fixed scenario, not a product menu. Record real identities, complete outputs, repeated conditions and dates; separate measurements, assigned traces, estimates and provider claims. Apply hard gates before preferences and cost calculations. KA-1 is complete when its evidence supports a decision, including a defensible rejection of every measured candidate.

33.9 Exercises and Part V checkpoint

1. Why does A’s six semantic passes in Table 33.1 mean 2/6 robust English cases rather than 6/18 independent scenarios? Explain why the same 750. is correct twice and wrong once.
2. Recalculate Example 33.2 at 5,000 monthly requests. Then double the output length to 400 tokens without changing the API rates or local assumptions. Find the new API per-request cost and break-even count.
3. A report uses the fixed 150 ms trace from Chapter 30 as A’s measured TTFT. Identify the evidence error and the timing that was actually measured for A.
4. Connect the following failures to the first layer you would investigate: current policy deleted by a filter; old KV reused after editing the prompt; empty EOS answer; valid answer rejected only for punctuation; many users waiting despite a fast single call.
5. A card says “open, 40M active, long context, chat.” State what is still unknown about total memory, license rights, architecture and usefulness on the updated lodging task.
6. A future candidate passes all twelve case-language pairs. List four additional kinds of evidence needed before calling it a production assistant, and explain why a successful KA-1 diagnostic alone is insufficient.

33.10 Reference answers

1. Two English cases pass on each of three repeats; repeat identity does not add task coverage. Both current questions require 750, while the historical question explicitly requires 600. A content string’s correctness depends on the evidence and question, not the digits alone.
2. At 5,000 requests, API cost is 20 dollars and local cost is 25. With 400 output tokens, API cost becomes $0.002+0.004=0.006$ dollars per request. Equality is $0.006R=20+0.001R$, so break-even is 4,000 requests. The unchanged local variable cost is an assumption, not a measured fact about longer answers.
3. It confuses an author-assigned runtime scenario with a measured checkpoint run. A’s actual recorded quantity is complete warm CPU generation-call time, including tracing and decoding; it is not streaming TTFT. Keep both records with their own labels.
4. Investigate data filtering/version governance; cache validity in the engine; generation/model behavior and stop records; evaluator format-versus-semantic rules; and server scheduling/capacity. These are starting hypotheses, to be tested rather than automatic diagnoses of every case.
5. Active count does not reveal resident total weights, cache or workspace. “Open” lacks an inspected license and modification materials. “Chat” can describe a wrapper; the architecture and context representation are still unspecified. Long-context capacity does not prove correct use of the new 750-yuan evidence.
6. Examples include realistic document lengths and citations, genuinely unused representative bilingual cases, independent human review, sustained load/tail latency, privacy/access controls, and recovery/operational ownership. Any four with a reason are appropriate. Twelve public compact diagnostics provide a narrow check, not coverage of all user distributions and deployment conditions.

33.11 References

• Liang et al., Holistic Evaluation of Language Models, 2023. Supports comparing explicit scenario/adaptation/metric combinations instead of treating a single ranking as product quality.
• Mitchell et al., Model Cards for Model Reporting, 2019. Supports the report’s identity, intended-use, evaluation and limitation fields.
• OSI, Open Source AI Definition 1.0. Identifies the definition used for the distinction between available weights and a complete open-source AI system.
• The official model, price, license, commercial-term and retention pages linked in the dated section support only those dated documentary observations. The full source locations and limitations are preserved in the Part V research ledger; none is a book-run benchmark.