Chapter 28 Mechanistic Interpretability: Decodability, Correlation, and Causality

28.1 Finding a color is not yet finding its cause

Suppose a model correctly answers “red” after a distant red-key clue. A researcher records an intermediate activation, trains a small reader on it, and finds that the reader can identify the color. Has the researcher found the mechanism that caused the answer?

Chapter 12 already separated a useful representation from a guarantee about what its coordinates mean. Here the gap is sharper: information may be present without being used by the output computation. A background feature might correlate with the clue, and an added reader might learn that shortcut. We need a question that can fail even when a visualization looks convincing: if we change this internal quantity while controlling the others, does the answer change in the predicted way?

We will analyze a fully specified two-coordinate model. Its values are author-assigned, and its outcomes follow from its equations. They are not measurements of MG-2, a discovered brain-like circuit, or proof about all LLMs. The point is that the entire causal claim can be checked, including the negative control.

28.2 A reader can succeed on an unused coordinate

Example 28.1 — Two correlated signals, one output path. The hidden row is $h=[r,b]$. In this constructed model only, r codes the key color: +1 for red, −1 for blue. The second coordinate b codes a background marker. The observational examples always pair red with marker +1 and blue with marker −1. Different filler texts do not change these two values.

The model’s output candidates are ordered red, blue. Its fixed output matrix is

$$W=\begin{bmatrix}2&-2\\0&0\end{bmatrix},\qquad z=hW=[2r,-2r].$$

The first row of W connects r to both candidate scores; the second row contributes zero from b. For $h=[1,1]$, logits are $[2,-2]$. For $h=[-1,-1]$, they are $[-2,2]$. Define the red-over-blue logit margin as red’s score minus blue’s: $m=z_{\mathrm{red}}-z_{\mathrm{blue}}=4r$. A positive margin favors red, a negative one blue. Its unit is a score difference, not a probability or an accuracy percentage.

A probe is an additional reader trained to predict a label from recorded activations while the original model stays fixed. In our paired distribution, the rule “read the sign of b” labels both colors perfectly. So does reading r. The first succeeds despite b having zero output weight. The success establishes decodability under the given pairing, not use by the original output computation.

For a learned probe, retain separate training and held-out examples, define labels independently, and compare with control tasks such as shuffled labels or simpler readers. A very expressive reader may memorize labels. Even a well-controlled successful probe still answers a reading question; it does not by itself answer a causal one. In our constructed counterexample, a test distribution with red and marker −1 immediately breaks the b shortcut.

28.3 A local sensitivity is a different question

An attribution assigns some measure of output influence to inputs or internal quantities. Different methods answer different questions. To see a local sensitivity without needing calculus, change r from 1 to 1.1 while leaving b at 1. The margin rises from 4 to 4.4: a 0.1 coordinate change caused a 0.4 score change. Their ratio is 4. Changing only b by 0.1 changes the margin by zero. For this linear model, those ratios hold at every point.

A gradient-based attribution uses such infinitesimal sensitivities in more general models. The operation matters: a local slope near the observed activation is not necessarily the effect of deleting an entire component or replacing it with another example’s value. A saturated nonlinear function can have a small local slope while a large replacement crosses into a different regime. Multiplying a gradient by an activation introduces another convention. Report the quantity, baseline and location instead of calling every colored heat map “importance.”

An attention weight is another recorded quantity. It describes a mixing coefficient at a particular layer/head/position. Values, output projections and later residual paths determine what that mixture ultimately does. A large coefficient can multiply an irrelevant or canceling vector. It supplies a candidate to investigate, not an automatic explanation of the final answer.

28.4 Replace a state and complete the computation

Activation patching runs the model on a baseline input, substitutes a selected internal activation from another run, then recomputes everything downstream of that substitution. The choice of donor, recipient, position, layer, replaced coordinates and output metric defines the experiment. Pasting a label into a result table does not perform an intervention.

Example 28.2 — A matched positive and negative control. Call red/+1 the clean run and blue/−1 the corrupted run. Here “corrupted” means the designated contrast, not damaged bytes. Both the clue and marker change in this contrast. We therefore test their coordinates separately. The candidate order, output matrix and margin stay fixed.

Table 28.1 — Complete derived intervention records

Run and operation	State at the readout	Logits [red, blue]	Red margin
Clean	[1, 1]	[2, −2]	4
Corrupted	[−1, −1]	[−2, 2]	−4
Restore clean r into corrupted	[1, −1]	[2, −2]	4
Restore clean b into corrupted	[−1, 1]	[−2, 2]	−4
Replace clean r with corrupted r	[−1, 1]	[−2, 2]	−4
Replace clean b with corrupted b	[1, −1]	[2, −2]	4

The third row restores the clean margin. The fourth uses the same donor/recipient pairing and replacement size, but a different coordinate; it restores nothing. The reverse replacements test disruption rather than rescue. Together they support a specific claim: in this defined readout, r mediates this color contrast and b does not. We already know W, so an independent multiplication can check every patched result.

A normalized rescue fraction is sometimes convenient. Here the clean–corrupted gap is $4-(-4)=8$. Restoring r gains $4-(-4)=8$, so rescue is $8/8=1$. Restoring b gains zero, so rescue is 0. Writing this convention generally,

$$R=\frac{m_{\mathrm{patched}}-m_{\mathrm{corrupted}}}{m_{\mathrm{clean}}-m_{\mathrm{corrupted}}}.$$

Each m is the same signed score margin in a different run. The denominator sets the reference improvement; it is not the number of examples. If the clean and corrupted margins coincide, this fraction is undefined. Other interventions can produce negative values or exceed 1. Clipping those values to the interval from 0 to 1 would conceal disruption or overshoot.

Observed pairing

1. A list of numbersClean [r,b]=[1,1]
2. Stored model numbersFixed W = [[2,−2],[0,0]]
3. A list of numbersMargin 4; both signs reveal red

Decodability follows the paired data; the zero weight blocks causal use of b here.

Restore r

1. A list of numbersCorrupted [−1,−1]
2. An operationReplace only r → [1,−1]
3. A list of numbersMargin −4 → 4; rescue 1

Same readout, downstream recomputed.

Restore b: matched control

1. A list of numbersCorrupted [−1,−1]
2. An operationReplace only b → [−1,1]
3. A list of numbersMargin −4 → −4; rescue 0

The replacement is equally large but does not affect this output.

Figure 28.1 — The matched ineffective patch is evidence too. Original book diagram · CC BY-SA 4.0.

Figure 28.1 holds the readout fixed while swapping one coordinate at a time. Read the unchanged zero row as carefully as the successful restoration: without that control, observational correlation could be mistaken for causal use.

28.5 A strong local result still has a scope

The patched states [1, −1] and [−1, 1] never occurred in the observational examples. They are deliberate counterfactual states. In this tiny linear model, the function is specified there, so we can analyze the consequences exactly. In a trained network, an unusual mixture may fall outside its ordinary activation distribution. An observed effect could then reflect disruption more broadly than the proposed mechanism.

A careful study varies clean/corrupted pairs, includes matched irrelevant replacements and reverse interventions, checks alternative corruption rules, and preserves cases that did not work. It specifies whether examples were chosen because the original model already answered correctly. If so, the conclusions apply to that selected subset; they cannot silently become an explanation of failures or all prompts. Position alignment matters too: changing wording can move tokens, and “same array index” may no longer identify the same role.

A rescue demonstrates sufficiency of the supplied activation to restore this metric in this recipient setting. A disruption supplies evidence of necessity in its tested setting. Neither automatically proves uniqueness. With redundant routes, removing one path may have little effect because another path still carries the signal. Conversely, a destructive replacement can lower performance without identifying the specific information normally used.

28.6 Reading intermediate predictions and tracing a circuit

A logit lens asks another reading question: what candidate scores result if an intermediate residual vector is passed through an output projection, with the declared final normalization? It is inexpensive evidence about how a representation relates to the output vocabulary. It need not be the prediction the full model will make after later layers.

Example 28.3 — Coordinates can change meaning between layers. Let an earlier row be $u=[b,r]$. A later operation swaps coordinates to form $h=[r,b]$, after which W produces the output. Applying W directly to u reads b. In our original paired examples, that lens appears to predict the color perfectly. In the counterfactual $u=[-1,1]$, the raw lens favors blue, while the swap followed by the actual readout favors red. The later transformation matters. Learned lenses attempt to account for such representational mismatch, but their training introduces another reader whose validation must be reported.

A circuit is a proposed collection of components and connections that implements a behavior under specified conditions. Our tiny candidate circuit is the r coordinate and its two nonzero connections to the logits. To test a circuit in a larger model, one might keep its paths while replacing others with a stated baseline and measure how well it preserves the full model’s behavior; then remove or alter proposed paths to test dependence. Path patching isolates selected connections rather than replacing an entire component output everywhere.

Three questions remain separate: does the proposed circuit reproduce the relevant behavior faithfully; does it omit additional influential routes; and can parts be removed without losing that reproduction? Consider two redundant Boolean routes whose outputs are combined by OR. With both equal to 1, deleting either one leaves the answer 1; deleting both changes it. A list of individually effective deletions can miss redundant mechanisms. Circuit results therefore need a task distribution, intervention definition and error criterion, not just a labeled diagram.

A model’s written explanation of its answer is generated text. It may help a user inspect an argument, but it is not a recording of these internal operations or proof of the model’s internal reasoning process. The experiments above manipulate arrays and follow their downstream effects. They do not infer a hidden transcript from fluent self-description.

The fixed states, matrix and outputs are in the shared Part V fixture, reproduced by the scalar program. Reading and evaluating these records is the required practice; patching an actual trained network is optional and has not been substituted with invented measurements. Having separated evidence about scores from evidence about causes, we can now return to a simpler runtime decision: how should scores become the next token?

28.7 Summary

Decodability, attribution and intervention answer different questions. A probe can read unused information; a local sensitivity depends on its baseline and operation; patching tests an explicitly defined replacement. A circuit claim adds claims about paths, coverage and redundancy. Strong evidence remains tied to the tested model, examples, locations and metric.

28.8 Exercises

1. In Example 28.1, set $h=[-1,1]$. Give the probe prediction from b, the actual logits and the selected color.
2. A different experiment has clean margin 6, corrupted margin 2 and patched margin 5. Calculate rescue. What if both reference margins are 2?
3. A paper chooses only prompts answered correctly, restores an activation and observes improvement. Which population does this support, and which two controls would strengthen the interpretation?
4. Change Example 28.3’s early row to $u=[1,-1]$. Compare the raw lens with the actual swap–readout computation.
5. A head has high attention to “red,” but replacing its output has no effect. Give two distinct possible explanations without declaring the head universally irrelevant.

28.9 Reference answers

1. The b probe predicts red. The model gives $[-2,2]$ and selects blue. This is the counterfactual that breaks the observational shortcut while leaving the true readout rule unchanged.
2. Rescue is $(5-2)/(6-2)=3/4=0.75$. With equal reference margins, the denominator is zero; report the raw margins and mark rescue undefined. Adding a tiny denominator would create a different, unstable statistic.
3. The result concerns the selected correctly answered prompts and the specified corruption. Matched irrelevant-coordinate replacements and reverse patches are useful controls; additional pairs or corruption rules test robustness. None silently extends the population to all failures.
4. The raw lens yields $[2,-2]$ and favors red. Swapping gives $[-1,1]$, whose actual readout is $[-2,2]$ and favors blue. An intermediate projection can disagree with the completed model.
5. Its value/output contribution may carry no useful color signal, or another route may redundantly preserve that signal. An ineffective replacement could also be poorly chosen. Further targeted tests must distinguish these explanations.

28.10 References

• Hewitt and Liang, Designing and Interpreting Probes with Control Tasks, 2019. Control tasks help distinguish representation evidence from the added reader’s capacity.
• Belrose et al., Eliciting Latent Predictions from Transformers with the Tuned Lens, 2023. Section 2 explains limitations of raw intermediate output projections.
• Meng et al., Locating and Editing Factual Associations in GPT, 2022. Section 2.1 specifies clean, corrupted and restored computations; our values are separate teaching constructions.
• Wang et al., Interpretability in the Wild, 2023. Sections 3–4 show path-based interventions and distinct circuit criteria in a restricted language task.