Chapter 49 Reliable Agents: Permissions, Budgets, Replay, and Recovery

The assistant has answered the lodging question and read mira’s order A-104. The employee now asks to cancel it. The application sends a cancellation request, waits, and receives nothing before its timeout. Should it send the request again?

A timeout tells us that the caller stopped waiting. It does not tell us whether the receiver changed the order. That gap, introduced in Primer-HTTP, becomes consequential as soon as a tool can write. A persuasive explanation of the failure cannot close the gap; we need evidence that survives it.

49.1 The first retry can be the first mistake

Example 49.1 — One intention, two side effects. All orders in this chapter are fictional. We add a local business simulator to the existing knowledge assistant, leaving Chapter 42’s immutable read-only snapshot intact. A fixture is a fixed set of test data and responses that lets us repeat the same conditions. The new fixture puts mira’s A-104 in tenant north, initially processing, at object version 1. A cancellation changes its status and records one effect. Imagine that this effect also represents a downstream cancellation notification: sending it twice would matter even if the final status still said cancelled.

The deliberately unsafe simulator increments the effect count whenever it accepts a cancellation. Send the same request twice: the first call changes the count from 0 to 1, and the second from 1 to 2. Looking only at the final status would hide the duplication. Our independent expected value is two effects because there were two accepted invocations under that unsafe rule.

The desired rule is different: repeated delivery of the same intended operation must not repeat its business effect. This is the relevant form of idempotency. It does not promise identical timing or an identical response string. It also does not mean every request containing A-104 must be treated as the same operation forever.

We give this particular intent a key and persist a receipt containing its result. The lookup is scoped by tenant and key; the receipt also binds the user and a digest of the action arguments. If the same key arrives with different arguments, the host reports idempotency_conflict. Otherwise it can return the saved result. Choosing a fresh key on every retry would destroy the relationship we are trying to preserve.

49.2 Make permission narrower than a sentence

Before the first write, the host checks a proposal equivalent to:

{
  "name": "cancel_order",
  "arguments": {"order_id": "A-104", "expected_version": 1}
}

The authenticated principal is supplied by the host, not by this object. In the simulator it is an immutable record containing tenant, user, role, audience, scopes and expiry. A scope names a limited permission; an audience names the service for which a credential is intended. The local record is a teaching substitute, not a signed credential or a production login system.

Table 49.1 — Approval and ordinary tool permission are separate.

Operation	Automatic under the teaching policy?	Required checks
Read mira’s own order	Yes, after the configured read permission	Valid principal, correct service, own tenant and owner
Read another user’s order	No	Rejected even if the model asks politely
Propose cancellation	May produce a proposal	Proposal alone performs no write
Approve cancellation	Only the designated approver role	Same tenant, exact subject and action, explicit decision and expiry
Execute cancellation	Only after approval	Write scope, matching valid approval, current object version and cancellable status
Send arbitrary HTTP or execute a shell command	No	No such capability is in the allowlist

Approval records a person’s permission for a concrete action. In the simulator, the approver has orders:approve; the acting user has orders:read:own and orders:cancel:own. Those are different roles. Approval binds the tenant, user, action digest and expiration. Approving A-104 version 1 does not approve A-105, another tenant’s A-104, or a newly changed version of the same order.

The version check protects against a stale plan. Suppose the order is shipped between proposal and execution. A previously reasonable cancellation should not blindly act on the changed object. The host rereads the state and rejects a mismatched version. A production service must enforce this check at its own write boundary; a client-side check followed by an unguarded remote write can race with another update.

“Read-only automatic” means the application has an established, narrowly scoped permission policy for those reads. It does not mean every readable resource can be shared with every model or third-party server. Chapter 53 follows those additional boundaries.

49.3 Commit the effect and its evidence together

The local simulator stores order rows, approvals, receipts, run checkpoints and events in SQLite. SQLite is a small database that can save several related updates as one transaction: either all these updates commit or none do. Here the cancellation effect and its receipt share one transaction.

First attempt

1. Application recordPENDING → CALLING; budget 3 → 2
2. An operationApproval + version checked
3. Outside the modelCommit cancellation + receipt
4. Application recordReply lost → UNCERTAIN

Business effect count is already 1.

Recovery after reopening

1. Application recordLoad UNCERTAIN + same key
2. An operationCALLING; budget 2 → 1
3. Text piecesReceipt found → DONE

Effect count stays 1. Replay has no CALLING edge and changes no database row.

Figure 49.1 — A lost response does not undo a commit. Original teaching diagram · CC BY-SA 4.0

Figure 49.1 places the lost reply after the joint commit. Its return path reads the durable receipt instead of repeating the business update.

Example 49.2 — Recover after the effect committed but its reply was lost. Start with effect count 0, a valid exact approval, three call units, and at most two attempts. The failure injector raises a timeout exception immediately after the cancellation and receipt commit. This is a deliberately placed fault, not a measured network outage. The record is nevertheless produced by executing the real local database path.

Table 49.2 — The complete lost-response trace.

Step	Persistent state or observation	Calls remaining	Effect count
Start	PENDING; proposal, approval ID and key saved	3	0
First attempt admitted	CALLING; deduct before the attempt	2	0
Transaction commits	Cancelled order and receipt both saved	2	1
Fault after commit	Caller records UNCERTAIN	2	1
Close and reopen the database	State, key and budget survive	2	1
Recovery attempt admitted	CALLING; same intent and key	1	1
Receipt returned	DONE, receipt_reused: true	1	1

The recovery attempt has a cost even though it does not create another business effect. That is why it consumes the second call unit. The independent check is not merely “the result says reused.” It inspects the persistent order and verifies that its effect count is still one, after closing and reopening the database.

Move the fault to just before commit. The first attempt now leaves zero effects and zero receipts. A permitted second attempt performs the operation and leaves one of each. These two tests locate the failure on opposite sides of the transaction boundary. They would expose a program that wrote the receipt too early or committed the effect without the receipt.

This atomicity is local to one database. If a real order service and the assistant’s log use different databases, writing a local receipt does not make the remote operation atomic. The remote service needs a durable idempotency contract or another reconciliation mechanism. “Exactly once” is not a property we can obtain by naming a client function retry_safely.

49.4 A budget must survive failure too

A retry policy needs several limits because each addresses a different failure. A per-attempt timeout bounds one wait; an attempt limit bounds repetition; a call budget bounds admitted operations; an overall deadline bounds the request’s allowed lifetime. Saving only the current stage and resetting the budget to three after every restart permits an unbounded sequence of “fresh” attempts.

Example 49.3 — Account for time and calls separately. These durations are authored logical-clock values, in seconds. The deadline is 1.00 second after start, the per-attempt timeout is 0.20 second, backoff is 0.05 second, and the initial call budget is three. The first attempt reaches its 0.20-second timeout. Its call unit was already deducted, leaving two. Before another attempt, the host waits 0.05 second. The second attempt succeeds in 0.10 second and consumes another unit.

Add the occupied intervals: $0.20+0.05+0.10=0.35$ seconds. The deadline has $1.00-0.35=0.65$ second left, while the call budget has $3-2=1$ unit left. Neither remaining quantity obliges the application to continue after success. Seconds and call units cannot be added together.

For an admitted attempt with remaining deadline allowance $D-t$, the allowed wait is:

$$w=\min(\tau,D-t).$$

Here $D$ is the deadline measured from the chosen clock origin, $t$ is the current time on that same clock, and $\tau$ is the per-attempt timeout, all in seconds. After the first timeout and backoff, $t=0.25$ and $w=\min(0.20,0.75)=0.20$. If the overall deadline were only 0.21 second, the 0.05-second backoff would no longer fit after the first timeout. The scheduler therefore makes no second attempt.

The logical scheduler demonstrates this arithmetic; it does not interrupt arbitrary Python code or prove a wall-clock service bound. The inherited HTTP client exercises actual transport timeouts. The cancellation simulator injects failures at exact commit locations so those cases are reproducible. A real worker needs an enforceable timeout in its transport or process boundary, and recovery must use a clock representation valid across restarts rather than a process-local elapsed-time origin.

Table 49.3 — Stop for the reason that actually applies.

Condition	Recorded result	Why a blind retry is inappropriate
Approval denied or expired	DENIED	Repetition cannot supply permission
No call units remain	STOPPED, call_budget	Restart must not replenish the allocation
Overall deadline passed	STOPPED, deadline	A spare call unit does not extend time
Two failed attempts	STOPPED, attempt_limit	Remaining budget may be reserved for other work
Object version or key conflicts	MANUAL_REVIEW	The intended action and current evidence no longer agree
Lost response with a valid saved receipt	Recover by reading that receipt	No new business write is required

Reading an existing receipt requires current read permission. It does not require a fresh cancellation approval because it performs no new cancellation. If the receipt is absent, execution must pass current write and approval checks. A run that has stopped at its deadline does not silently resume; a human can reconcile its recorded outcome through the permitted read path.

49.5 Replay displays evidence; recovery may act

A trace records externally inspectable events: state, step order, time, selected action, relevant versions, approval reference and result. It is not a transcript of a model’s private internal reasoning, and a fluent explanation is not a substitute for a receipt. The trace should avoid credentials and unnecessary personal text.

Replay in this project reads saved events and displays the original sequence. The replay test checks that the database’s change count does not increase. It makes no model call and no tool write. Reproducing a screen from old records says what was recorded then; it does not establish that today’s source or service would respond identically.

Recovery reopens persistent state and continues from a valid checkpoint. It can execute another operation, so it must preserve budgets and recheck permission. The local simulator has one worker; concurrent worker leasing, distributed transactions and remote credential refresh are outside its tested scope. A checkpoint is a durable application record, not a vague note that the task was “almost finished.”

Compensation is a new action intended to address an earlier effect. It is not reversal of time. After a cancellation, making a new booking could have a different price or fail because the room is gone. Our compensation handler records COMPENSATION_REVIEW, returns no_automatic_inverse, and performs zero business writes. That is an explicit recovery policy for this operation, not a missing automatic “undo.” Any proposed replacement booking would require its own authorization and failure handling.

49.6 Where independent-agent protocols fit

A2A describes communication among independent agents, including tasks and produced artifacts. MCP describes access to tools and context. These responsibilities can complement each other; substituting one name for the other does not solve approvals or recovery. The current specification’s versioned details are in the online note. KA-6 uses neither an A2A connection nor a second agent: the known task fits one host-managed workflow, and additional agents would add coordination and failure states that need justification.

49.7 Practice and the boundary of KA-6

reliable_runtime.py provides the local store and executor; source_skill.py provides the source-check package integration. The practice runner saves normal, denied, exhausted and interrupted traces. The Notebook closes and reopens the actual SQLite file before recovering, then compares effect counts. It also preserves the unsafe two-effect baseline in a separate temporary database.

KA-6 now has a reusable source check, scoped reads, explicitly approved local writes, finite budgets, dependency identity and recoverable failure evidence. The teaching principal, logical scheduler and single-database transaction are deliberately visible limitations. No actual user order, payment or remote service is changed.

49.8 Summary

The important question after a timeout is what committed, not how confident the assistant sounds. Exact approvals bind permission to an intended action; scoped idempotency keys and durable receipts connect retries to that intent. Budgets and checkpoints must survive failures. Replay reads history, recovery can act, and compensation is a separate potentially fallible action. These controls make specific failure paths inspectable; the next chapter asks how to evaluate the entire system rather than counting successful final answers alone.

49.9 Exercises

1. Repeat Example 49.3 with a 0.30-second deadline, 0.20-second first timeout, 0.05-second backoff and a second operation needing 0.10 second. How much wait can the second attempt receive, and does it succeed under the stated rule?
2. After the lost-response case, recovery generates a new idempotency key. Explain why this breaks the intended guarantee even if this simulator’s version check happens to prevent a second effect.
3. Approval names A-104 version 1. A new request asks for A-105 version 1 using the same approval. Trace which permission check fails. Does reading both orders successfully change the answer?
4. A dashboard reconstructs a successful cancellation from a saved trace. An operator calls this “re-executing the workflow.” Explain the distinction and give one observable test.
5. The cancellation committed, but a promised replacement room is unavailable. Should compensation mark the original order as processing? Describe a truthful state and the next responsible action.

49.10 Reference answers

1. At the second attempt, 0.25 second has elapsed. The deadline leaves $0.30-0.25=0.05$ second, so the allowed wait is $\min(0.20,0.05)=0.05$ second. The operation needs 0.10 second and times out. Two calls have been consumed, but a remaining unit cannot extend the deadline.
2. A new key represents a new intent to the receipt lookup and loses the saved-result path. In this particular fixture, the old expected version then conflicts with the changed order, so the host stops for manual review. That secondary check is not a general idempotency guarantee: another side effect or a permissive service could still repeat. Preserve the original key and argument binding.
3. The action digest differs because the target differs; the approval does not match. Read permission establishes neither write permission nor approval for the new target. No business mutation should occur.
4. Replay reads a prior record; re-execution calls current code against current state and may cause effects. Record database change count and tool-call count before and after: this replay changes neither. A successful replay cannot prove that a fresh execution would succeed.
5. No. That would falsify the committed business state and suggest a booking exists when it does not. Keep the original cancellation and receipt, record compensation as unresolved/manual review, and have the responsible operator obtain and approve a concrete alternative. The replacement action needs its own receipt and failure path.

49.11 References

• MCP authorization, 2026-07-28. Scope handling and security considerations explain why identity, audience and authorization remain explicit host/service obligations.

• A2A specification and A2A and MCP. These locate independent-agent task exchange relative to tool access; no A2A implementation is required for the book’s local workflow.

• NIST AI RMF 1.0, GOVERN 2. These connect technical recovery to assigned human responsibility. The database examples and all budget values in this chapter are original teaching constructions.

• SQLite atomic commit. The introduction explains the local all-or-none transaction property; a remote effect is outside this database boundary.