Chapter 14 Attention: Where Should the Current Output Look?

14.1 The next output may need an earlier state

The encoder in Chapter 13 produced a useful color state and then lost it before reaching its final summary. The decoder received only [0,0], which could not distinguish the red source from the blue source. Making the decoder more elaborate cannot recreate a distinction absent from its only source representation.

Suppose we keep all three encoder states. Before writing a color, the decoder can combine them one way; before writing the object, it can combine them another way. The source is still read by a recurrent encoder and the output still written by a recurrent decoder. We are changing what source information each output step can access.

This is the motivation for attention in recurrent translation: calculate a context for the current output from stored source representations instead of relying solely on one fixed summary. Bahdanau and colleagues formulated such a model with a learned alignment function. We will calculate the mechanism using a simpler, explicitly declared scoring rule. Bahdanau et al., 2015, §3

14.2 Give one decoder position three possible places to read

Example 14.1 — Read for a color from the computed source states. Reuse the exact source [红色, 钥匙, 。], embeddings, and ReLU encoder from Chapter 13. With $a=\ln2\approx0.693147$, the stored states in source order are:

$$h_1=[a,0],\qquad h_2=[0,a],\qquad h_3=[0,0].$$

These states were calculated from weights. They are not the original input embeddings: for example, the key embedding was [-a,a], whereas its state is [0,a]. A stored state summarizes a computation involving its prefix; it is not necessarily an isolated word vector.

Now supply a decoder-state probe $q_1=[1,0]$. The letter $q$ stands for a query: the vector with which this output position compares the available source states. In our recurrent skeleton, a query is the previous decoder state. The two query values used in this chapter are deliberately supplied probes, not a learned decoder trajectory or values generated by Chapter 13’s probability table. This lets us inspect the read operation without inventing unprovided decoder weights.

Both query and source states have two coordinates. Define the score as their unscaled dot product: multiply corresponding coordinates and add. It is not cosine similarity; we do not divide by lengths. It is also not the scaled dot product introduced in the next chapter. This score is one of the alternatives described by Luong and colleagues; Bahdanau’s original learned additive score is different. We use the same unscaled rule throughout this chapter and its code. Luong et al., 2015, §3.1, equation (8)

Table 14.1 — From comparisons to source-position weights.

Source position	State	Dot product with [1,0]	Exponential	Normalized weight
1: color	[a, 0]	1 × a + 0 × 0 = a	2	2/4 = 1/2
2: key	[0, a]	1 × 0 + 0 × a = 0	1	1/4
3: full stop	[0, 0]	0	1	1/4

The score row is [a,0,0]. Chapter 6’s Softmax turns it into nonnegative weights summing to one. Because $e^a=2$ and $e^0=1$, the denominator is $2+1+1=4$. Dividing by three would give the average exponential, not a probability distribution. A zero score has exponential 1, so it does not receive zero weight.

These probabilities distribute a read across the three source positions, not across candidate output words. A weight of 1/2 for the color state does not mean a 50% chance of outputting red. To obtain the context, multiply each entire source row by its weight, then add matching coordinates:

$$\begin{aligned} c_1&=\tfrac12[a,0]+\tfrac14[0,a]+\tfrac14[0,0]\\ &=[a/2,a/4]\approx[0.346574,0.173287]. \end{aligned}$$

The first coordinate receives $a/2$ from the color state and zero from the others. The second receives $a/4$ from the key state. The full-stop state contributes a zero vector, yet its positive weight still occupies part of the normalization total. It affects how much weight the other states receive.

This weighted sum is the current context vector. It has two coordinates, just like each state being summed; it does not have three coordinates just because there are three positions. In this toy computation, every coordinate of the result must lie between the smallest and largest corresponding source coordinates: nonnegative weights adding to one form a weighted average. The result cannot create an arbitrary new value outside those coordinate ranges.

14.3 Change the output position, and the context changes

Example 14.2 — Read for an object from the same memory. Keep all source states fixed and supply a second query $q_2=[0,1]$. Its scores are now [0,a,0]: the dot product selects the second coordinate instead of the first. Exponentiating gives [1,2,1]; dividing by the same total 4 gives weights [1/4,1/2,1/4]. Therefore:

$$c_2=\tfrac14[a,0]+\tfrac12[0,a]+\tfrac14[0,0] =[a/4,a/2]\approx[0.173287,0.346574].$$

The first read weights the color state most; the second weights the key state most. Nothing in the source memory changed. The current query changed the combination returned. This is what dynamic access means here, without implying a conscious search or a reliable decision about what matters.

Read 1: q = [1,0]

1. A list of numbersColor [a,0] × 1/2
2. A list of numbersKey [0,a] × 1/4
3. A list of numbersFull stop [0,0] × 1/4
4. A list of numbersSum → [a/2, a/4]

Scores [a,0,0] → exponentials [2,1,1] → divide by 4.

Read 2: q = [0,1]

1. A list of numbersColor [a,0] × 1/4
2. A list of numbersKey [0,a] × 1/2
3. A list of numbersFull stop [0,0] × 1/4
4. A list of numbersSum → [a/4, a/2]

Scores [0,a,0] → exponentials [1,2,1] → divide by 4.

Figure 14.1 — Change the query; keep every source state fixed. Original computed teaching diagram, CC BY-SA 4.0. Queries are supplied probes; source states are calculated from the declared encoder.

Figure 14.1 keeps the source order visible while exchanging the first two weights. A soft correspondence between target position and source positions is often called alignment. It need not select one position exclusively, follow the diagonal, or reproduce a human word-to-word translation alignment. The weights arise from the declared score; whether that read helps a learned decoder is a separate question.

After carrying out the operation, we can write its general form. Let $n$ be the number of source states and $d$ their width; the query also has width $d$. For target position $t$ and source position $j$:

$$e_{t,j}=q_t\cdot h_j,\qquad \alpha_{t,j}=\frac{\exp(e_{t,j})}{\sum_{k=1}^{n}\exp(e_{t,k})},\qquad c_t=\sum_{j=1}^{n}\alpha_{t,j}h_j.$$

The scalar $e_{t,j}$ is a comparison score; its letter is not itself the exponential constant. The index $k$ runs over all source positions included in the denominator. The scalar $\alpha_{t,j}$ is that position’s weight. Summing scaled rows gives $c_t$, a row of width $d$. For Example 14.1, $n=3$ and $d=2$: there are three scores and weights, but only two context coordinates. Substitution reproduces Table 14.1 and [a/2,a/4].

This operation is cross-attention because the query comes from the target-side decoder while the states being read come from the source-side encoder. Here we directly use source states both for comparison and as the values being averaged. Separate learned projections and the full Q/K/V notation are deferred to Chapter 15.

14.4 Give the context to the decoder, rather than treating it as an answer

The context supplies information for a prediction; it is not yet that prediction. Keeping Chapter 13’s recurrent ordering, set $q_t=s_{t-1}$, obtain the attention context, and then update the decoder:

$$s_t=F(s_{t-1},E_y(y_{t-1}),c_t),\qquad p_t=\operatorname{softmax}(s_tW_o+b_o).$$

The old decoder state is $s_{t-1}$; the previous target embedding is $E_y(y_{t-1})$; the new ingredient is a context $c_t$ that changes with the read. $F$, $W_o$, and $b_o$ have the roles explained in Chapter 13. For example, if each of the three input rows has width two, concatenating them gives six input coordinates for the recurrent update. It does not give six output-word probabilities. The final Softmax still normalizes over the output vocabulary, separately from the attention Softmax over source positions.

Teacher forcing still supplies the previous correct target during training. Generation still supplies the previous chosen target. Access to the source states has not removed that prefix difference. Nor does the supplied pair of query probes establish what a trained decoder would actually say at those positions.

The extra path does make the toy encoder’s earlier distinction accessible. On the blue source, the states are [0,0], [0,a], [0,0]. With the first query, all scores are zero, so all weights are 1/3 and the context is [0,a/3]. This differs from the red source’s [a/2,a/4], even though their final encoder states were equal. The decoder now has a source-dependent distinction it could use. Whether it learns an appropriate output rule is not established by this calculation alone.

14.5 Delete a position, and specify what stays fixed

The color state had the largest weight in Example 14.1. Can we therefore declare it the main cause of any subsequent output? That conclusion needs an output quantity and an intervention, not just a ranking.

Example 14.3 — Remove one stored state before the read. Freeze the first query and the numerical values of all remaining source states. Remove one memory position entirely from both the score list and the weighted sum, then normalize over the remaining positions. This is not setting the removed score to zero: a zero score would still contribute exponential 1.

To measure an effect, define a simple downstream scalar readout: first context coordinate plus twice the second. Writing these coordinates as $c_{t,1}$ and $c_{t,2}$, define $r=c_{t,1}+2c_{t,2}$. This readout is also an authored diagnostic, not a learned translation score. Before deletion, it gives $a/2+2(a/4)=a$.

Table 14.2 — Frozen-memory deletion with the first query. Weights remain listed in original source order; zero marks an excluded position.

Removed position	Weights: color, key, full stop	New context	Readout r
None	[1/2, 1/4, 1/4]	[a/2, a/4]	a
1: color	[0, 1/2, 1/2]	[0, a/2]	a
2: key	[2/3, 0, 1/3]	[2a/3, 0]	2a/3
3: full stop	[2/3, 1/3, 0]	[2a/3, a/3]	4a/3

Removing color leaves two zero scores, so the remaining weights are each 1/2. The context changes, but the chosen readout does not: doubling the larger second coordinate exactly replaces the lost first-coordinate contribution. Removing key leaves exponentials 2 and 1, so the denominator becomes three and the readout falls to $2a/3$. Removing the zero-valued full-stop state also changes the normalization, raising the readout to $4a/3$.

Thus the largest attention weight need not identify the removal that changes this output quantity most. A different readout, such as the first context coordinate alone, would give different effects. Attention weights report how this read combines states; causal claims about a particular output require tracing the downstream computation and defining the change being tested. This is a limited conclusion, not a claim that attention is never useful evidence. Jain and Wallace, 2019, §4; Wiegreffe and Pinter, 2019, §2

Deleting an original input token is another intervention. It may change the states at subsequent positions, so we must run the encoder again. For example, delete the source key token, retaining [红色, 。]. The first state is [a,0]. At the full stop, adding [0,-a] gives [a,-a]; ReLU produces [a,0]. Both remaining states now equal [a,0]. Holding the first query fixed, both scores equal a, each receives weight 1/2, and the context is [a,0]. The readout is a, unlike the $2a/3$ obtained by deleting only the frozen key state.

Table 14.3 — Original-input deletion with encoder recomputation. The query and readout stay fixed, so this is still not a full decoder rollout.

Deleted source token	Recomputed states	Context	Readout r
Color	[0,a]; [0,0]	[0,a/2]	a
Key	[a,0]; [a,0]	[a,0]	a
Full stop	[a,0]; [0,a]	[2a/3,a/3]	4a/3

Some results coincide with Table 14.2, but the key result does not. “Deleting a word changed the answer” is incomplete evidence unless we know whether later states, the query, and later generated tokens were recomputed. A correlation between high weights and apparent relevance may be worth investigating; it is not an intervention result. Even a properly recorded intervention establishes the effect under its specific conditions, not a universal explanation of the model.

There must also be at least one available state. If all positions are excluded, there is no positive normalization total and no defined weighted read under this rule. The practice code reports an error rather than manufacturing an all-zero probability distribution.

14.6 More access still has storage and sequential costs

Keeping all source states adds a direct route from each stored representation to each target read. In the old arrangement, an early source contribution had to pass through every remaining encoder update to reach the final context. The new read can use an earlier saved state without that particular detour. Training gradients can also travel through the read to that state. This does not remove all long paths or guarantee that useful representations and alignments will be learned.

The memory cost is visible even in this example. The fixed context stores two numbers. Three source states store six. With an explicitly assumed eight bytes per number, that is 16 versus 48 bytes of logical activation storage, excluding parameters, containers, decoder state, and training intermediates. For $n$ source positions and width $d$, source-state storage has $nd$ numbers. These are counts under assumptions, not measured process memory.

If each of four output steps scores all three source positions, there are twelve dot products; each two-coordinate dot product contains two scalar multiplications and an addition. The full operation also exponentiates, normalizes, and computes weighted sums. Longer input increases the work of every read.

Source dependency

1. A list of numbersh₀ → h₁ → h₂ → h₃
2. An operationSave all three states

An early saved state has a direct route to the read, without passing through the remaining encoder updates.

Known query

1. A list of numbersqₜ = sₜ₋₁
2. An operationqₜ·h₁, qₜ·h₂, qₜ·h₃
3. A list of numbersNormalize and sum → cₜ

The three comparisons may run in parallel; they all require the current query first.

Target dependency

1. An operationOld state + previous token + cₜ → sₜ
2. A list of numbersNext query uses sₜ

Teacher forcing supplies tokens, not future hidden states. During generation, the next input token must also be selected.

Figure 14.2 — Parallel comparisons sit inside a sequential recurrent calculation. Original computed teaching diagram, CC BY-SA 4.0. Queries are supplied probes; source states are calculated from the declared encoder.

Figure 14.2 marks what still has to wait. For a known query, comparisons against separate source states and parts of the weighted sum can be computed in parallel. But the recurrent encoder still needs three successive state updates here, and the decoder still needs four. Even under teacher forcing, the next decoder state depends on the preceding one. During generation there is the additional dependency on the previous chosen token. Attention has not turned the entire recurrent sequence into parallel updates.

Access also remains bounded by what the application supplied. If the updated 750 CNY travel policy never entered the encoder, attention cannot retrieve it from outside the system. A source state’s relevance score does not verify a policy version or authorize an order operation.

14.7 Part III checkpoint: identify the path before proposing a fix

Table 14.4 — Four diagnoses that should not be merged into “poor memory.”

Observed evidence	Established problem	What would address that path?	What remains unproved?
Same final two input tokens, different required colors	Fixed-window input collision	Supply more context or carry computed state	That the model will use the added clue correctly
A local loss signal shrinks across many dependency edges	Weak backward sensitivity	Better state paths and training choices; gates may help	Reliable retention at arbitrary distance
Training gets red at step 2; generation gets blue	Reference/generated prefix mismatch	Evaluate behavior under generated prefixes	That every early error will cascade
Different sources have the same final vector	Fixed-summary collision in this encoder	Give target steps access to earlier states	Correct learned alignment and output selection

The checkpoint is practical: point to inputs, states, and shared parameters in Figure 11.1; recover the sum −0.75 in Table 12.1; locate the first changed input in Table 13.5; and calculate the weights and context in Table 14.1. Then use Tables 14.2–14.3 to explain why an attention picture is not automatically a causal explanation. The visible reference answers across the four chapters provide the reasoning for each step.

The shared practice is code/part-iii/run.py, with fixed values and expected results in data/part-iii/. notebooks/part-iii/02-seq2seq-attention.ipynb executes the trace, compares both deletion procedures, and optionally checks attention using NumPy. The standard-library path requires only the Python primer. All core results remain visible here; no account, GPU, paid API, or running notebook is needed to understand the mechanism.

14.8 Summary

Cross-attention calculates a source-position distribution from a current decoder query, then returns a weighted source-state vector. Our two queries produce different contexts from unchanged memory, recovering access to a distinction that the final summary lost. The context still needs a decoder output rule, and an attention weight does not by itself explain that rule’s result.

The source and target recurrences still impose sequential dependencies. Chapter 15 asks whether positions within a sequence can interact directly, instead of passing information only through recurrent neighbors. That changes another information path: self-attention. Learned projections, scaling, and the later Transformer structure belong to that next argument.

14.9 Exercises

1. Read a new query. Use query [1,1] with the original three source states. Compute all scores, weights, context coordinates, and the diagnostic readout. Keep answers as fractions of a where possible.
2. Diagnose a denominator. A learner calculates [2/3,1/3,1/3] for Example 14.1 because there are three source positions. Identify the error. What weights result from query [0,0] instead?
3. Compare two removals. Derive the two different contexts for “delete key”: first remove only its stored state, then delete its input token and re-encode. Explain which remaining state changes.
4. Test an explanation. The color state has weight 1/2. A report says it must therefore have the largest causal effect on the final response. Give a numerical counterexample from the chapter and name the output quantity and intervention that it actually tests.
5. Count costs and dependencies. Suppose there are 100 source states of width two and four target steps. At eight bytes per number, how much logical source-state storage is needed? How many dot-product comparisons? Which recurrent dependencies remain even with known training targets?

14.10 Reference answers

1. Scores are [a,a,0], exponentials [2,2,1], and the total is five. Weights are [2/5,2/5,1/5]. The context is [2a/5,2a/5], and the readout is $2a/5+2(2a/5)=6a/5$. The three weights sum to one; the two context coordinates need not.
2. The denominator must sum the exponentials, giving four, not count positions. The proposed weights sum to 4/3 and are not a distribution. A zero query gives three zero scores and three exponentials of 1, so in that different case the denominator is three and the weights are all 1/3.
3. Removing the frozen key state leaves [a,0] and [0,0], with weights 2/3 and 1/3 and context [2a/3,0]. Removing the input key makes the later full-stop state [a,0], since the missing key no longer cancels the color coordinate. Both remaining states are now [a,0], so their equal-weight context is [a,0].
4. Under frozen-memory deletion and the declared scalar readout, removing the highest-weight color state leaves the readout at a, while removing key lowers it to 2a/3. This refutes the automatic ranking claim for that quantity and intervention. It is not a measurement of translated-word accuracy or of an entire generated response.
5. The states contain $100\times2=200$ numbers, occupying $200\times8=1600$ bytes under the stated assumption. Four reads require $100\times4=400$ dot products. The encoder still updates source states in order, and the decoder still updates target states in order; known target tokens do not make recurrent hidden states independent. These counts omit other tensors and implementation overhead.

14.11 References

• Bahdanau, Cho, and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate (2015). Section 3 explains distinct per-target contexts and a recurrent decoder that reads source annotations. Its encoder and additive score differ from this toy construction.
• Luong, Pham, and Manning, Effective Approaches to Attention-based Neural Machine Translation (2015). Section 3.1 supplies the unscaled dot-product scoring alternative. We use that scoring rule without claiming to reproduce its complete decoder order.
• Jain and Wallace, Attention Is Not Explanation (2019). Section 4 compares attention with other importance measures and tests changed attention distributions on specific tasks.
• Wiegreffe and Pinter, Attention Is Not Not Explanation (2019). Section 2 examines the assumptions behind such explanatory claims and interventions; read it alongside the preceding paper and our explicitly defined deletions.