Chapter 56 LLMOps II: Caching, Resilience, SLOs, and Feedback

Mira asks the same lodging question twice. Reusing the first checked answer looks sensible: why retrieve and verify the same three lines again? Then the policy date changes, another tenant asks an apparently identical question, and the dependency stops responding. A speed improvement has become a correctness and availability problem.

Chapter 55 identified the versions behind each result. That gives us a way to ask a more precise question: under which conditions may an old computation be reused, and what should the service do when a new computation cannot finish? The answer must include what the user receives and how the operator counts it.

56.1 Reuse a result only when its conditions still hold

The simplest response cache maps a question to a previously computed final result. With the key Beijing lodging, a 2025 answer of 600 could be returned for a 2026 question even though the retrieval code and corpus are correct. The missing date in the key bypasses the very check the system was designed to perform.

A cache key identifies the computation being reused. In our implementation it combines the authenticated tenant, user and scopes; all four request fields; and the release-manifest digest. The digest covers the data, lexical processing, context, answer/schema/Skill code and relevant configuration. Exact question bytes must match; case or spacing changes may cause a harmless miss. This conservative choice saves fewer computations than a broad normalization rule, but does not silently equate two different questions.

Example 56.1 — Same words, different eligible answers. Both policies remain in the same corpus. Request A asks Beijing lodging for 2025-12-31 and should receive 600 from v1. Request B uses the same words for 2026-01-01 and should receive 750 from v2. Corpus identity alone cannot distinguish them, because the data bytes are unchanged. The date must also participate. Changing the language changes the returned quotation, and changing the authorized principal can change permitted records, so those conditions participate too.

The service authenticates each incoming read and checks job ownership before returning a stored result. It checks authorization before a response-cache lookup and does not cache order reads. A cache hit is a shortcut through computation, not a shortcut through permission. It also does not update a user’s persistent memory or teach the model anything.

Table 56.1 — Four different things called a cache

Cache	What is retained	What a hit requires	What it does not supply
Within-request KV	Numerical attention state for earlier positions	Compatible continuation and model computation	A reusable final policy answer
Prefix cache	Computed state for a matching input prefix	Matching IDs, positions and computation conditions	Permission or merely similar meaning
Response cache	A final checked result	Matching request, access and processing scope	Automatic knowledge refresh
Semantic cache	A result associated with a sufficiently similar query representation	A similarity rule plus all required access/date/version checks	Proof that the questions have the same answer

The first two were introduced in Chapter 31 and save model computation. The latter two act at the application level. Our default uses only exact response caching; no neural model state is involved.

56.2 Expiry is a time rule, not a truth detector

Example 56.2 — Compute the last valid instant. A response is inserted at monotonic time 10 seconds with a time to live (TTL) of 30 seconds. It expires at $10+30=40$ seconds. At 39.999 seconds, a matching entry is usable. At exactly 40, it is not. This is an assigned clock example checked by the implementation, not a measured service lifetime.

Let $t_i$ be insertion time, $\tau$ the TTL and $t$ the lookup time, all in seconds. Time validity requires:

$$t<t_i+\tau.$$

The strict inequality makes the boundary unambiguous. Substituting gives $39.999<10+30$, true, and $40<10+30$, false. A hit also requires the correct key and current authorization; this formula describes only the time condition.

Suppose a new authorized policy becomes effective at time 20. A 30-second TTL would still allow the old entry until 40 unless something else changed. The repair is a new corpus/release identity and a compatible index, making the old key unreachable for new requests, or an explicit invalidation. TTL limits how long an entry may be reused under unchanged identity. It does not discover that a fact has changed.

The local cache holds at most 64 entries and evicts the least recently used when full. It stores copies so one caller cannot mutate another caller’s cached object. Expired entries are rejected at lookup and periodically removed. A release switch produces a different namespace even if the question text stays fixed. Restoring an old release deliberately restores old processing conditions; it is not a way to conceal a required policy update. The operator must distinguish a bad application configuration from an authoritative new document.

56.3 Similar questions can still require incompatible answers

A semantic cache can improve hit rate when users paraphrase. It first turns queries into representations, searches for a close stored query and may reuse that query’s answer. This is attractive when exact matching misses synonyms. It also introduces a new decision whose errors are answer errors, not just missed optimization opportunities.

Example 56.3 — A very close vector, a wrong amount. Assign the historical-date query vector $u=[1,0]$ and current-date query vector $v=[0.99,0.01]$. These are hand-chosen teaching coordinates, not learned embeddings. Their dot product is 0.99. Their lengths are 1 and $\sqrt{0.99^2+0.01^2}=\sqrt{0.9802}$, so cosine similarity is approximately 0.999949. A threshold of 0.99 admits the match, although the required amounts are 600 and 750.

Raising the threshold to 0.999 still admits it. Similarity describes these coordinates; it does not prove date equivalence. Even a perfect lexical paraphrase must respect tenant, time, version and task constraints. For this small service, the evidence does not justify an additional semantic acceptance rule, so semantic caching stays disabled. A future experiment would need fixed false-match cases, independent answer labels and a measured tradeoff between saved work and wrong reuse.

56.4 Control admission before work overwhelms the service

A cache helps repeated requests, but a stream of distinct questions still consumes work. Chapter 54 limits active answer processes to two. We also need to limit how quickly one principal can start new work. Otherwise, one caller can occupy every available slot repeatedly.

A token bucket keeps a numerical allowance; this “token” is a request permit, not a language-model token. The bucket has a maximum capacity, refills at a stated rate and spends one permit for each admitted attempt. An attempt rejected by the bucket consumes no permit, but remains a visible rejection. The service’s defaults are four permits and two permits per second. The smaller numbers below make the arithmetic easy to inspect.

Example 56.4 — Follow a two-permit bucket. Give the bucket capacity 2, initially full, and refill rate 1 permit per second. Three requests arrive at time 0. The first leaves 1 permit, the second leaves 0, and the third is rejected. At time 0.5, half a permit has accumulated, so another request is rejected. At time 1, one whole permit is available and is spent. Continue as follows; times are assigned seconds, not a traffic measurement.

Table 56.2 — Refill first, then decide

Arrival time	Permits before spending	Admit?	Permits afterward
0, first request	2	Yes	1
0, second	1	Yes	0
0, third	0	No	0
0.5	0.5	No	0.5
1	1	Yes	0
1.5	0.5	No	0.5
2	1	Yes	0

If $b$ permits remained at the previous observation, $r$ is the refill rate, $\Delta t$ elapsed seconds and $C$ capacity, the amount before spending is:

$$b'=\min(C,b+r\Delta t).$$

Here $r\Delta t$ has units of permits. From time 0.5 to 1, $b=0.5$, $r=1$ and $\Delta t=0.5$, giving $b'=1$; admission then subtracts one. The minimum prevents ten idle seconds from creating more than capacity 2. The bucket allows a short burst and controls a sustained rate. It is not a reservation guaranteeing an idle worker: a permitted request can still receive 503 if the global concurrency limit is full.

The actual HTTP tests separately produce 429 for rate limiting and 503 for occupied capacity. Retrying immediately would add load to the condition that caused the failure. The response provides a wait hint; the user’s next attempt remains subject to current capacity and its own deadline. The service does not interpret a retry hint as a promise of success.

56.5 Stop calling a dependency that keeps failing

A transient failure can justify a bounded retry. A persistent outage makes repeated retries wasteful. The local fault runner raises a controlled dependency-unavailable exception while serving real HTTP jobs. Each job may attempt the read-only computation twice, with a 50 ms wait between attempts. That injected failure is real software execution under an authored condition; it is not a measured outage of an external provider.

A circuit breaker uses observed failures to decide whether to make another call. In the closed state calls may pass. After the configured failure threshold, the breaker becomes open and rejects calls immediately. After a cooldown, half-open permits a limited probe. A successful probe closes it; a failed probe opens it again. The breaker protects the caller while somebody repairs the dependency. It does not perform the repair.

Example 56.5 — Count failed requests, not attempts. In the hand-checkable fixture, two consecutive failed requests open the breaker and the cooldown is five seconds. At time 0, one request fails after its allowed attempts; the failure count is 1. At time 1, another fails; the count reaches 2 and the breaker opens. At time 2, a new request receives fallback without contacting the dependency. At time 6, exactly five seconds after opening, one probe is allowed. It succeeds, resets the count and closes the breaker. At time 7, ordinary calls can pass again.

Assigned clock: two failed requests, five-second cooldown

1. Application record0 s: failure count 1
2. Application record1 s: count 2 → open
3. Application record2 s: fallback; zero attempts
4. Application record6 s: one probe succeeds → closed

Time since opening is 6 − 1 = 5 s. A second half-open probe is blocked while the first is active.

Actual local HTTP drill: cooldown 0.25 s

1. Application recordRetry recovers: good
2. Application recordTwo failed jobs + open rejection: 3 bad
3. Application recordProbe recovers: good

Five eligible jobs: 2 good / 5 = 40%. Earlier errors remain after recovery; this is injected smoke traffic, not production availability.

Figure 56.1 — Recovery changes future calls; it does not erase earlier failures. Original book diagram · CC BY-SA 4.0.

Figure 56.1 keeps two clocks apart: the fixed explanatory timeline uses five seconds; the actual local smoke drill shortens cooldown to 0.25 seconds. In the actual record, two jobs exhaust their attempts, the next job contains circuit_open and no attempt event, and a later probe returns 750. Another record recovers on its second attempt, without opening the breaker. A separate delayed job reaches its deadline. These distinct paths test different controls.

Here fallback is an explicit temporary-unavailability result with the option to prepare a note for a person. It does not return an expired policy as though it were current. A lower-cost model, lexical search or last-known result could be another fallback, but only under a separately evaluated contract with visible limitations. The service already uses lexical retrieval, so calling that same path “a new fallback model” would add no capability.

Cancellation is not a failed dependency response. The defect found in Chapter 54 matters here: counting a user stop as a dependency failure could open the breaker during perfectly healthy operation. The regression now requires cancellation to bypass dependency-failure accounting. Monitoring needs correct event meaning before it needs a prettier dashboard.

56.6 Define the promise before calculating its percentage

“How often did the service work?” has no denominator until we define a request and a good outcome. Our teaching service-level indicator (SLI) measures server-observed completion behavior. An eligible request has valid input and authenticated identity. A good one reaches an answered, abstained or refused terminal state within three seconds. Authorized requests rejected for capacity or rate count as bad. Unauthenticated or malformed requests are reported separately and excluded. Explicit cancellations are also reported separately and excluded from this particular completion indicator.

This choice does not claim that every abstention is a good answer. It measures whether the service completed according to its response contract. The missed synonym in Chapter 54 can be good for this SLI and bad for answer-quality evaluation. Nor does a server terminal event prove that a browser received it. Delivery needs a client-side indicator or external probe. These qualifications belong beside the number.

A service-level objective (SLO) sets a target for an indicator over a declared window. We assign 95% for this exercise, over a half-open window from 0 up to but excluding 60 seconds. This is a teaching target, not a measured customer requirement, a contractual service-level agreement (SLA) or a production recommendation.

Example 56.6 — Count the window. The fixed data contains 22 observations: 18 timely completions, two eligible errors, one explicit cancellation and one unauthenticated request. The last two are separately counted. Thus 20 observations enter the denominator, and 18 enter the numerator:

$$\mathrm{SLI}=\frac{18}{20}=0.90.$$

The denominator is not 18, because eligible failures must remain visible. It is not 22, because the indicator deliberately excludes two separately reported categories. With a 95% target, the allowed bad fraction is $1-0.95=0.05$. Across 20 eligible requests, the error budget is $20\times0.05=1$ bad request. We observed two, so the objective was missed by one request. The budget is not permission to deliberately harm that one user; it is a way to prioritize reliability work when the aggregate objective is threatened.

A burn ratio compares the observed bad fraction with the permitted fraction:

$$B=\frac{2/20}{1-0.95}=2.$$

The numerator is the observed fraction of bad eligible requests, 0.10. The denominator is the allowed fraction, 0.05. Both are dimensionless, so $B=2$ means twice the target’s allowed bad-event rate in this window. It does not say which change caused the failures or how long the same rate will persist.

With a 99% target, the same observations yield a budget of $20\times0.01=0.2$ and a burn ratio of 10. A fractional budget is a valid aggregate allowance, not a fraction of an executable request. With so few requests, one failure already exceeds it. With no eligible requests, the SLI and burn ratio are undefined; returning 100% would invent evidence.

The local monitor reports a session window and retains at most 1,000 observations. A job still in flight has not emitted its terminal observation, so a live snapshot is provisional; settle those jobs before interpreting a completed window. If older rows are dropped, it marks the window incomplete; an operator must not present the retained subset as complete session availability. Diagnostics rotate separately, and traces can be sampled in larger systems. None of those storage policies changes the definition of the desired denominator.

56.7 An alert asks for investigation, not belief in a cause

For the demonstration, the alert rule requires at least five eligible observations and a burn ratio strictly greater than 2. The fixed 18/20 example has burn ratio exactly 2, so it misses the objective without firing this particular alert. That is intentional: meeting an alert threshold and meeting an SLO are different predicates. The actual injected-fault sequence contains two successful completions and three errors, so its five-request indicator is $2/5=40\%$ and its burn ratio is $(3/5)/0.05=12$; it does fire.

A practical alert policy also needs observation windows, a recipient, a response and a clearing rule. Here the local operator checks the failed traces, stops the injected failure, waits for the cooldown probe and confirms a successful request. The saved session ratio still includes earlier errors; one successful probe does not erase the window. Longer-running systems often compare short and long burn windows to balance fast detection against isolated noise. This tiny drill does not choose those production parameters for the reader.

If latency improves after a release, caching, a changed request mix, fewer users or a recovered dependency may explain it. Monitoring observes what happened in the collected distribution. To attribute improvement to the release, compare suitably controlled conditions, preserve denominators and inspect other changes. A timestamped correlation is a useful lead, not a causal proof.

56.8 Turn a report into a reviewed test, not an automatic lesson

The interface offers three feedback categories: wrong source, missing answer and difficult to use. It stores the category and a request reference under the authenticated principal. It does not collect arbitrary free text, credentials or a proposed replacement policy. Reports enter quarantine: a review queue whose contents cannot change the corpus, answer rules or model parameters.

Example 56.7 — A report is not an answer key. A synthetic missing-answer report is submitted. Before review, the regression list remains empty. The maintainer checks the source’s inclusive 2026-01-01 start date, writes a separate synthetic case expecting 750 from v2 and records the rationale. That case then enters the reviewed list, and an actual check returns 750. Deleting the original subject’s report removes the user-linked feedback; the reviewed fixture contains no user identity. This is an explicit author review and a new regression proposal, not automatic training or automatic deployment.

The service limits pending reports, expires them after seven days with a periodic cleanup pass and supports subject deletion. The local process keeps them in memory, so restarting ends that queue. A persistent feedback system would need access-controlled storage, deletion propagation and an operator workflow before making the same promise across restarts. A useful regression must survive separately as reviewed, appropriately de-identified material with provenance and a version.

Voluntary feedback has selection bias: reporters need not represent all users. Suppose 100 requests occurred, ten users reported a problem and eight of those reports concerned a wrong source. Then $8/10=80\%$ is the fraction among reports, not the wrong-source rate among all requests. Even $8/100=8\%$ is only the observed report fraction; unreported errors and mistaken reports prevent it from being an accuracy estimate. Systematic labeled samples and targeted user research complement feedback rather than being replaced by it.

KA-10’s operational increment is now executable through the shared service policies, fault runner and Notebook. Fixed logical clocks make boundary arithmetic reproducible; actual HTTP records demonstrate the separate runtime paths. The risk and runbooks preserve remaining limitations, including the known lexical gap, single-service-instance operation and lack of production availability evidence. The graduation chapter will use those boundaries to decide what has actually been delivered.

56.9 Summary

Caches reuse computations only within explicit identity, permission, date and expiry conditions. Semantic similarity adds a fallible acceptance decision. Admission limits, deadlines, bounded retries and a circuit breaker protect different boundaries; fallback must remain truthful about what is unavailable. An SLO starts with a defined population, outcome and window, while an alert only signals a condition worth investigating. Feedback becomes reliable maintenance evidence through review and versioned regressions, not through unexamined storage or automatic training.

56.10 Exercises

1. An entry is inserted at 12 seconds with a 20-second TTL. Classify lookups at 31.999 and 32. If policy data changes at time 15, does the TTL by itself prevent stale reuse at time 20?
2. A bucket has capacity 3, refill rate 2 permits/s and is empty at time 4. Requests arrive at 4.25, 4.5 and 5. Give the pre-spend amount and decision each time.
3. A window contains 48 timely completions, two eligible overload rejections, three cancellations and two unauthenticated attempts. Under this chapter’s indicator and a 98% target, calculate SLI, budget and burn ratio. What would change if the product chose to include cancellations?
4. A breaker opens after two failed requests. Each request is allowed two attempts. Explain why counting attempts can open it earlier than intended, and why a cancelled request should not count as a dependency failure.
5. A new release coincides with fewer complaints and lower median latency. Give a plausible alternative explanation for each observation and propose stronger evidence.
6. A report says “750 is wrong; replace it with 900.” Describe the permissible route from this report to a possible source or regression update, including what must remain unchanged before review.

56.11 Reference answers

1. Expiry is $12+20=32$ seconds: the first lookup is time-valid, the second is not. TTL alone still permits reuse at time 20. Change the relevant data/release identity and compatible index, or explicitly invalidate; also check access and request scope.
2. At 4.25, there are $0+2\times0.25=0.5$ permits: reject and keep 0.5. At 4.5, another 0.5 arrives: admit and leave 0. At 5, another 1 arrives: admit and leave 0. The rejection does not spend the partial permit.
3. The denominator is 50, so SLI is $48/50=96\%$. Allowed bad events are $50\times0.02=1$, observed bad events are 2, and burn is $(2/50)/0.02=2$. Including cancellations as bad gives 53 eligible observations, $48/53\approx90.566\%$, budget 1.06 and burn $5/1.06\approx4.717$. That is a different declared indicator, not a correction to arithmetic under the original one.
4. The first request’s two failed attempts could reach a threshold of two before a second request even exists. Keep the unit of counting explicit. A cancellation says the caller stopped work; it is not evidence that the dependency failed. Separate it and release any held probe without inventing a failure.
5. Fewer users might submit complaints, or the reporting UI might be harder to find; a higher cache-hit share or easier workload might reduce latency. Inspect exposure and reporting rates, sample labeled outcomes, compare equivalent request mixes and versions, and use an appropriate control before claiming causation.
6. Quarantine the report. Do not change the authoritative corpus, cache identity, answer rules or model merely because a user proposed a number. An authorized policy owner must establish whether 900 is valid and for which interval; a maintainer reviews provenance and creates a versioned source/regression change, preserves historical cases, evaluates the candidate and follows the release gate. Rejected or unsupported reports remain distinct from accepted labels.

56.12 References

• Google SRE Workbook: Implementing SLOs, SLI specification/implementation and error-budget policy. Supports explicit measurement boundaries and documented targets; the book’s numbers are assigned.
• Google SRE Workbook: Alerting on SLOs, burn-rate and multiple-window sections. Explains why alerting criteria require more than a single success percentage.
• Google SRE: Handling Overload, client throttling and utilization. Supports protecting capacity; the token-bucket fixture is an original simplified policy.
• Microsoft: Circuit Breaker pattern, solution and retry distinction. Supports closed/open/half-open behavior; this project’s consecutive-request counter is stated separately.
• OWASP LLM10:2025 Unbounded Consumption, resource, timeout and queued-action controls. Motivates bounded work without certifying resistance to hostile traffic.
• NIST AI 600-1, Generative AI Profile, MANAGE 4.1–4.3, printed pp.44–45. Supports reviewed feedback, issue handling and documented recovery responsibility.