Chapter 55 LLMOps I: Versions, Traces, Releases, and Rollbacks

Yesterday the assistant answered 750. After a configuration change, the same question receives 600 with a perfectly readable citation. The maintainer’s first note says “model unchanged.” That observation is true and nearly useless: the running service is extractive, and the change occurred before answer production. The host replaced the user’s 2026 date with a 2025 date, then correctly found and quoted the wrong applicable policy.

Chapter 54 made each request visible. We now need to identify what produced it and decide which changes may receive traffic. LLMOps names the continuing engineering work around an LLM-based application: maintaining versions, evidence, releases and operations. The mechanisms also matter when a particular path uses no neural model. They attach to the system’s behavior, not to a model’s brand name.

55.1 Record the dependencies that can change an answer

Saving only the final text is enough to reread it, but not enough to explain it. Saving the question and model name still omits which policy was supplied, how it was selected, which instructions surrounded it and which checks accepted the result.

Example 55.1 — The version record for the same lodging request. Keep the input from Example 54.1: Beijing lodging, 2026-09-14, travel, English. The service assigns the admitted request an immutable release manifest. A manifest is a structured inventory of the components and configuration chosen together. “Immutable” here means the request keeps its own snapshot; switching the active candidate does not edit that snapshot.

Table 55.1 — What the manifest identifies and why it matters

Component	Actual local identity	A change that would matter
Code and runtime	File SHA-256 digests; Python 3.12.10; jsonschema 4.25.1	Different guards, parsing or date handling
Answer/model producer	Extractive adapter; neural weights absent	Replacing extraction with a model requires a new contract evaluation
Tokenization	The inherited lexical tokenizer and vocabulary bytes; no neural tokenizer	Different query terms can change ranking
Prompt and context	KA-2 content and context-contract files; actual packed evidence in the trace	Omitted scope or a changed budget can remove necessary evidence
Data	The unchanged line-addressable v1/v2 corpus and its digest	Altered text, authority or effective interval
Index	BM25 recipe, card bytes, candidate order A–E, stable tie rule	A different candidate or score can enter context
Output schema	ka-answer-v1 schema bytes	Different accepted types or required citations
Tools and permissions	Tenant-aware read store; separate read-only MCP implementation	Changed ownership or permitted operations
Skill	Source-verification 1.0.0 plus reviewed resource digests	A changed helper can silently accept an invalid source
Release configuration and UI	Date-override setting, response-cache TTL, UI/resource bytes	Wrong date interpretation or misleading presentation

SHA-256 computes a fixed-length digest from bytes. The inventory stores the full 64 hexadecimal characters, not only a short display prefix. Matching a reviewed digest helps detect changed content; it neither proves that the content is safe nor identifies who approved it. A release name such as rc-a is a convenient label. The manifest digest identifies the whole selected inventory and configuration.

In the local default, neural weights, neural tokenizer and sampling seed are explicitly absent, rather than filled with invented values. Lexical arithmetic still uses binary64; scores use the inherited rounding/tie convention. If a neural adapter is introduced, its weights or immutable provider snapshot, tokenizer, prompt template, decoding options, seed where supported and execution environment must join this record. A stable API model name alone may not identify provider-side bytes.

The index in this project is rebuilt from small fixed cards, so its identity is the recipe plus its input bytes. A large persisted index would also need the built artifact and its compatibility with the corpus and embedding model. Two index files named “current” need not refer to the same knowledge. Conversely, keeping archived v1 in the corpus is correct: it is still needed for historical questions.

Dependencies of the 750 answer

1. Outside the modelCorpus + lexical tokenizer → index
2. Application recordOriginal date + v2 → context
3. An operationProducer + schema + Skill → answer

Changing the date changes context even when the corpus, model field and schema do not change. The manifest identifies the whole combination.

Admission across the rollback boundary

1. Application recordBefore: faulty date → 600
2. An operationSwitch active candidate back to A
3. Application recordNew job: A → 750; old job keeps 600

Rollback affects new admission. Cancel or drain old work separately; restoring a configuration does not reverse an order effect.

Figure 55.1 — Restore compatible dependencies and preserve each request’s version. Original book diagram · CC BY-SA 4.0.

Figure 55.1 shows why versions form a dependency graph. The index depends on documents and tokenization; packed context depends on the request and selected evidence; the final answer depends on both production and validation. A rollback must restore a compatible combination, not merely change the first version number somebody notices.

55.2 Read a trace as a record of work

A trace connects the operations belonging to one request. A span records one bounded operation, including its beginning, end and relationship to another span. Our root span covers the admitted job; child spans cover retrieval/composition and the source Skill. Events record changes such as acceptance, cache lookup, an attempted call and a terminal result. This follows the useful relationships described by OpenTelemetry, while using a small local JSON format. It is not an OTLP exporter or an SDK conformance claim.

Example 55.2 — Separate the elapsed time from the work it contains. Suppose a constructed root span runs from 0 to 220 ms. Its retrieval/composition child runs from 70 to 100 ms and its Skill child from 100 to 200 ms. The root duration is 220 ms, retrieval takes 30 ms and the Skill takes 100 ms. Adding root and children gives 350 ms, but double-counts time already inside the root. The 90 ms not inside either child consists of 70 ms before them and 20 ms afterward. Those intervals need more instrumentation before they can be attributed to specific causes.

If two children overlap, even summing their durations can exceed the time their parent spent waiting. A trace’s nesting expresses relationships, not an instruction to add every number. Actual run records use measured monotonic times and preserve IDs linking child spans to the request. Reading that trace can locate a slow stage; it cannot establish that the model’s internal reasoning caused the delay.

The everyday diagnostic file stores IDs, release identity, events and spans, while omitting credentials, question text and answer bodies. The public experiment pack separately retains full inputs and results because every account and policy is synthetic. This separation is intentional. Copying production prompts into a public “reproducibility” folder would create a new privacy problem, and hashing a short private question would not make it anonymous against guessing.

Sampling also changes what a trace collection can show. If only one request in ten is retained, absence of a failed trace does not prove absence of failed requests. The service’s counters and retention limits therefore need their own accounting; Chapter 56 uses explicitly defined windows instead of treating a folder of traces as the whole population.

55.3 Four operations that are often confused with replay

A colleague asks to “replay the bad request.” Before doing anything, determine what result they need. In this example, four operations have different effects.

Table 55.2 — What repeats, and what may change

Operation	What is read or executed	Result and effect boundary
Read the saved trace	Existing events, input and output bytes	Shows what was recorded; no new answer or order action
Recompute a read-only request	Run retrieval and extraction again under a declared manifest	Produces a new observation; compare it with the old one
Call a model again	Rebuild the input and invoke the model/runtime	Even pinned settings do not universally guarantee identical output
Recover a business action	Consult durable intent, approval and receipt before deciding a next action	May require a controlled write; never inferred from trace playback

The local experiment reads the failed trace twice and obtains the same record, with zero new model calls and zero order effects. It also makes a new HTTP request after rollback, which returns 750. That is a new request under a restored configuration, not evidence that the historical 600 response never happened.

A seed specifies an input to a random process, not a universal reproducibility warranty. Model/runtime changes, execution order, floating-point operations or unavailable provider versions can prevent exact regeneration. Even if the distribution is preserved, a repeated stochastic call need not repeat a particular draw. State which level is promised: identical stored bytes, comparable conditions or an independently repeated outcome. Our deterministic extractive replay is deliberately narrower than a general-model reproducibility claim.

For a cancelled order with a lost response, the appropriate next step is Chapter 49’s receipt-based recovery. Its fresh capstone run again finds one effect after reopening the database; reading its recorded events produces no additional effect. Neither “run every span again” nor “roll back the answer model” is a safe substitute.

55.4 Make the failed candidate fail before it receives ordinary traffic

Example 55.3 — A release gate with an independent answer key. Candidate A keeps the user’s date and uses a 30-second response-cache TTL. Candidate B keeps the date and changes only that TTL to 10 seconds. The deliberately faulty candidate overrides every question date with 2025-09-14. The source corpus and executable code remain identical. This is a configuration experiment, not a new model-training comparison.

Before running candidates, the author specified five cases per language from the source intervals. Both English and Chinese use the same criterion and corresponding questions. Expected values are in expected-v1.json, separate from the implementation.

Table 55.3 — Actual candidate evaluation on the same ten cases

Case, repeated in both languages	Required result	A	B	Faulty date override
Current date: 2026-09-14	750, v2	Pass	Pass	Fail: 600, v1
First day of v2: 2026-01-01	750, v2	Pass	Pass	Fail: 600, v1
Last day of v1: 2025-12-31	600, v1	Pass	Pass	Pass on this case
Absent topic: volcano	Abstain without a citation	Pass	Pass	Pass
Approval rule in 2026	Supported approval-policy quotation	Pass	Pass	Fail: no eligible source
Total across both languages	Ten case outcomes	10/10	10/10	4/10

The current-date failure is particularly informative. The faulty answer quotes v1 accurately and passes checks against the internally substituted date. Its JSON can be valid and its citation exact while it fails the user’s task. A gate that checks only parsing or quotation matching misses this defect. The independent expectation must retain the original requested date.

Here the evaluation gate requires every one of these narrow regression cases to pass. The release entry also checks the candidate’s bytes and binds the report to that manifest digest: a passing A report cannot authorize B. The larger release checklist additionally requires current software/security checks, an available previous candidate, a rollback rehearsal and a named operator. The ten-case report does not stand in for those obligations or for population-level answer accuracy.

Represent three decisions by Boolean values: $I$ says the selected inventory matches, $Q$ says the required task regression passes, and $O$ says the operational checklist has been completed. Then the local release decision is:

$$L=I\land Q\land O.$$

The symbol $\land$ means all conditions must hold. It is not an average. If the faulty candidate has $I=1$, $Q=0$ and $O=1$, then $L=0$. Adding a thousand easy passing cases does not excuse a violated hard requirement. This is the human release contract; the small switch method implements the manifest/report checks, while the operator remains responsible for the recorded checklist.

55.5 Limit exposure, then practice getting back

A passing gate removes known regressions, but cannot establish that every future request will behave well. A canary gives a candidate a limited, time-bounded share of work while keeping a control. Compare the two under relevant conditions, and decide in advance what stops the experiment. Otherwise, “a few users seemed happy” can become an unreviewed full rollout.

Example 55.4 — The local cohort and rollback drill. The runner numbers ten requests from 0 through 9. Candidate B receives numbers divisible by ten; A receives the others. Only 0 qualifies, so B receives $1/10=10\%$ of this assigned cohort and A receives $9/10=90\%$. All ten actually traverse the local HTTP service and return 750. The deterministic route demonstrates cohort selection, not randomized evidence that B is safer or faster. With only one candidate request, a meaningful reliability estimate is unavailable.

The runner then switches the active configuration to B. To exercise recovery, it explicitly bypasses the failed gate for the faulty candidate inside the isolated local drill. This override is recorded as forced_local_fault_drill; it is not available through the user-facing API. A current-date request actually returns 600. The operator restores A, and the next request returns 750 with v2. The journal retains every switch, its previous and new manifest, and its time.

A request admitted under the faulty candidate before rollback is deliberately kept in flight. It retains that manifest and still returns its historical-date result afterward. Rollback changes admission for new requests; it cannot silently rewrite already admitted work. In an incident where continuing old work is unacceptable, the operator must separately cancel or drain it and tell affected users what happened. That policy also needs tests.

Our drill rolls back configuration with unchanged executable code and no data migration. It does not demonstrate undoing a destructive schema migration, restoring deleted source records or reversing an external order cancellation. Such changes require compatibility planning, retained data and, where applicable, business compensation. Calling all of them “rollback” would hide the conditions that make recovery possible.

55.6 Keep the candidate and the evidence together

The three candidate manifests, gate rows, failed and recovered HTTP traces, cohort and journal are saved under data/part-ix/. The runner is code/part-ix/run_records.py. It reuses earlier mechanisms and writes a new Part IX record rather than altering earlier model or baseline results. The Notebook inspects these relationships and executes small read-only checks; the runbook gives the full local rehearsal command.

Maintainers must capture candidates after reviewing their source bytes. Regenerating a manifest from whatever happens to be on disk and declaring it approved would erase the distinction between inventory and review. This project’s routine tests check stored evidence hashes; intentional regeneration is a separate author command. A future code change invalidates the recorded candidate and requires new evidence.

KA-10 now has version relationships, request traces, a replay distinction, a task-specific gate and an actual local rollback. Its remaining responsibility is operational: when work repeats or a dependency fails, how much capacity and waiting time will the service spend before it protects itself and informs the user?

55.7 Summary

A release is a compatible set of code, data, processing rules and user-facing behavior. A manifest names that set; a trace records work performed under it; an independent gate checks the user’s requirement. Saved-record replay, recomputation, model re-invocation and side-effect recovery are different operations. A local canary and rollback make routing and restoration inspectable, while retaining in-flight ownership and the limits of configuration-only recovery.

55.8 Exercises

1. A team retains model weights and seed but changes its tokenizer and context template. Which comparison claims remain justified, and what must be added before a repeat run can be interpreted?
2. A root span lasts 500 ms. Two children each last 300 ms and overlap for 200 ms, with both entirely inside the root. How much time is covered by at least one child, and how much by neither? Why is 1,100 ms not the request duration?
3. A candidate passes 999 easy questions but answers the current lodging question using v1. The contract makes current-date source correctness mandatory. Can a 99.9% overall score admit it? Explain the appropriate gate.
4. A request was admitted just before rollback, and an order cancellation was committed before its response was lost. Which of those facts can a model/configuration rollback undo? Give the separate next actions.
5. A local canary receives one of ten assigned requests and passes it. Give two justified conclusions and two conclusions the evidence does not support.

55.9 Reference answers

1. The saved old output remains an authentic record, but the new call no longer uses identical input processing. Record tokenizer bytes/version, the final input/template, context selection and execution settings. Treat the new run as a changed-condition comparison; the shared seed alone does not restore the old conditions.
2. The union of child intervals is $300+300-200=400$ ms. Another $500-400=100$ ms lies outside both children. Summing root and both children double-counts nested and overlapping time; the request still lasts 500 ms.
3. No. The mandatory source/date requirement failed, so its gate is false regardless of the other 999 outcomes. Preserve the failure, repair or reject the candidate, and rerun the same relevant cases. Report the broad score separately without allowing it to hide the hard failure.
4. Neither fact is undone automatically. The admitted request retains its snapshot unless separately cancelled or drained. The committed order action requires receipt-based recovery and possibly a separately authorized compensation; blindly repeating it or replaying spans risks another effect.
5. It supports that the implemented routing assigned one request to the candidate and that this request passed its stated criterion. It does not establish production reliability or a causal quality/latency improvement over the control. Exposure size, selection and workload must be stated.

55.10 References

• OpenTelemetry Tracing API, SpanContext, Span, Span Creation and IsRecording. Supports the span relationships and the distinction between retained trace samples and complete counters; the local format is intentionally smaller.
• Google SRE Workbook: Canarying Releases, definition and release-engineering principles. Motivates limited candidate exposure and compatible version management; its production observations are not measurements of this book’s service.
• NIST AI 600-1, Generative AI Profile, MS-2.3 and MS-2.5, printed pp.30–31. Supports documenting evaluation conditions and resisting claims extrapolated from narrow demonstrations.