Chapter 12 Why RNNs Forget: BPTT, LSTM/GRU, and Contextual Representations

12.1 Can the last mistake change an early computation?

The long blue-key example in Chapter 11 failed even though a nonzero trace of blue reached the last hidden state. That was a statement about a forward calculation with fixed weights. Training raises a different question: if the final prediction is wrong, how strongly does its loss respond to a change in an early calculation? A route that exists in the forward graph may carry a weak training signal backward.

Chapter 7 supplied the needed rule: along a sequence of dependencies, multiply the local rates of change; when separate routes reach the same parameter, add their contributions. We will apply that rule to three readings before considering a longer sequence. No additional calculus course is assumed. The gradient tables below are exact consequences of chosen factors, not measured training curves.

12.2 Follow one error backward, then add the shared uses

Example 12.1 — One weight appears at every reading. Keep just the positive color coordinate from Chapter 11. The scalar inputs are 1, 0, 0; the initial state is zero. The input weight is 1 and the recurrent weight is 0.5. The forward states are therefore 1, 0.5, 0.25. We now introduce a separate diagnostic objective: make the last state approach the scalar target 1. This is not Chapter 11’s output Softmax or a claim that hidden coordinates normally have supervised color targets.

The last error is $0.25-1=-0.75$. Squaring makes either sign of error costly; multiplying by one half simplifies the rate of change without changing which state minimizes the loss:

$$L=\tfrac12(h_3-1)^2=\tfrac12(-0.75)^2=0.28125.$$

Here $L$ is one scalar loss, and $h_3$ is now the single coordinate in this diagnostic. A tiny increase in the last state reduces the loss at a local rate of −0.75. To see where that rate comes from, change the state by a small amount $\delta$. Expanding the square changes the loss by $-0.75\delta+\tfrac12\delta^2$. Dividing by the change and making it smaller leaves −0.75. We write this rate as $\partial L/\partial h_3=-0.75$.

A change of 0.01 in the second state changes the third state by only 0.005, because their connection multiplies by 0.5. The loss is therefore half as sensitive to the second state as to the third: $-0.75\times0.5=-0.375$. Crossing the preceding connection multiplies by 0.5 again, giving $-0.375\times0.5=-0.1875$ for the first state. The minus signs describe the direction that decreases the loss, not a negative amount of stored information.

Table 12.1 — The backward signal and each direct use of the shared weight.

Reading	Previous state	New state	Loss rate with respect to new state	Contribution from this use of the weight
1	0	1	−0.1875	−0.1875 × 0 = 0
2	1	0.5	−0.375	−0.375 × 1 = −0.375
3	0.5	0.25	−0.75	−0.75 × 0.5 = −0.375

Why multiply by the previous state in the last column? At a particular use, the recurrent multiplication is “weight times previous state.” If we hold that previous state fixed and change the weight by 0.01, the local result changes by 0.01 times the previous state. Multiplying this local rate by the loss rate gives that use’s contribution. Earlier uses already account for how the previous state itself depends on the same weight; counting that dependence again inside the last use would double-count it.

All three uses belong to one parameter, so their contributions add: $0-0.375-0.375=-0.75$. We must neither update three independent weights nor average by three just because there are three readings. An average would belong to the definition of a loss averaged across targets; this objective has one terminal target.

Writing $w$ for the shared recurrent weight, the general scalar calculation is:

$$h_t=x_t+wh_{t-1},\qquad \frac{\partial L}{\partial w} =\sum_{t=1}^{T}\frac{\partial L}{\partial h_t}\,h_{t-1}.$$

The inputs $x_t$ are supplied scalars; $T$ is the number of readings. Each loss rate includes the later steps that depend on that state. This formula is for this linear scalar recurrence with a fixed initial state; nonlinear activations add their local rates. Substituting Table 12.1 gives −0.75 again.

There is an independent check. For these inputs and a zero initial state, the last state is $w^2$. Thus $L(w)=\tfrac12(w^2-1)^2$, whose rate is $2w(w^2-1)$. At $w=0.5$ it is also −0.75. With a learning rate of 0.1, one gradient step gives $w_{\text{new}}=0.5-0.1(-0.75)=0.575$. A new forward pass ends at 0.330625, and the loss decreases to 0.2240314453125. The old hidden states are not silently retained as if they had been computed with this new weight.

Forward

1. A list of numbersh₁ = 1
2. A list of numbersh₂ = 0.5
3. A list of numbersh₃ = 0.25
4. An operationLoss = 0.28125

h₁ → h₂ → h₃: each state passes through the same w = 0.5.

Backward

1. A list of numbersAt h₃: −0.75
2. A list of numbersAt h₂: −0.375
3. A list of numbersAt h₁: −0.1875

h₃ → h₂ → h₁: multiply the loss rate by 0.5 at each edge.

Shared parameter

1. An operationUses: 0 − 0.375 − 0.375
2. A list of numbersOne gradient for w: −0.75
3. Stored model numbersUpdated w = 0.575

Sum the contributions → update the shared w once, with learning rate 0.1.

Figure 12.1 — Multiply backward along the path; add across uses of one weight. Original teaching diagram and supplied or derived values, CC BY-SA 4.0; no paper figure reproduced.

Figure 12.1 follows the states forward and the loss rates backward. Applying backpropagation to this time-unfolded computation is backpropagation through time (BPTT). It means storing or recomputing the needed forward activations, traversing their dependencies backward, and summing contributions for shared parameters. It does not mean sending information into the real past. Pascanu et al., 2013, §2

12.3 Distance multiplies the problem

In Example 12.1, a signal crossing two edges was multiplied by one half twice. Across eight such edges, the multiplier is $0.5^8=1/256$. Across sixteen it is $0.5^{16}=1/65536$. This describes how a small perturbation is transmitted, rather than the state’s absolute size.

Example 12.2 — Change the local factor, keeping distance fixed. Suppose every edge instead has local rate 1.5. Crossing two edges now multiplies a signal by $1.5\times1.5=2.25$. The same repeated operation that can shrink a signal can also amplify it.

Table 12.2 — Signal multiplier versus number of edges crossed.

Distance: edges	Factor 0.5 at each edge	Factor 1.5 at each edge
0	1	1
1	0.5	1.5
2	0.25	2.25
4	0.0625	5.0625
8	0.00390625	25.62890625
16	0.0000152587890625	656.8408355712891

Distance zero has multiplier 1 because no transformation has yet changed the signal. The columns have no physical unit; they are ratios of small changes. They were calculated from powers, not fitted to training measurements. An initial loss rate of −0.75 would be multiplied by the relevant entry; the table itself does not include that rate.

Repeated shrinkage is the central intuition behind a vanishing gradient. The final loss gives an early parameter use little signal relative to nearer uses. Repeated amplification can produce an exploding gradient, making an ordinary-sized learning rate yield a very large parameter update. Neither requires literal zero or infinity. The original analyses of long dependencies examine conditions under which these effects arise; they do not say that every recurrent sequence behaves the same way. Bengio et al., 1994, §4

A real hidden state has several coordinates, so many routes can mix and add. Local rates also depend on the activation. ReLU has rate 1 above zero and 0 below zero; at the corner, an implementation chooses a convention. Another common activation, tanh, smoothly compresses any scalar to between −1 and 1. Its definition is $\tanh(a)=(e^a-e^{-a})/(e^a+e^{-a})$. For instance, tanh of 0 is 0 and tanh of 2 is about 0.964. Near large positive or negative inputs it changes very little: its local rate is $1-\tanh^2(a)$. At 2 this is only about 0.071. Large weights alone therefore do not determine the gradient; activation rates and the routes through the network also matter. Hochreiter and Schmidhuber, 1997, §3

Two practical responses address different costs. Gradient clipping limits the size of a large gradient before the optimizer uses it; it does not reconstruct a tiny signal lost along a long path. Truncated BPTT limits how far backward a training calculation follows time dependencies. A program can carry the numerical state into the next chunk while stopping gradients at the chunk boundary. It has retained a value, but has not assigned credit through the entire preceding sequence. This saves training memory and computation at the price of an approximate gradient. Dive into Deep Learning, §9.7

12.4 Decide how much to keep and how much to write

Halving every state at every reading is inflexible. A filler word and a new decisive clue use the same rule. We would like a route that can retain a coordinate strongly on one step and replace it on another. This motivates a gate: a computed multiplier that controls how much of a numerical value passes through a particular branch. It is not a yes/no judgment made by an agent.

Recall the sigmoid from Chapter 7: $\sigma(a)=1/(1+e^{-a})$. It maps finite scores to values strictly between 0 and 1. A gate value of 0.9 passes 90% of a value; 0.2 passes 20%. We will also discuss exact 0 and 1 as useful limiting settings. In a trained network, a gate’s score is calculated from the current input and previous hidden state using learned weights and a bias. The gate’s activation changes between steps even when those weights stay fixed.

12.4.1 A separate cell path in LSTM

Example 12.3 — Retain part of a cell and add a selected update. Suppose a scalar cell contains 0.8. Supply a retention multiplier 0.9, an input multiplier 0.2, and a candidate update 0.5. Retaining the old contribution gives $0.9\times0.8=0.72$. Admitting part of the candidate gives $0.2\times0.5=0.10$. Add them to obtain a new cell value of 0.82. An output multiplier 0.75 then exposes $0.75\tanh(0.82)\approx0.506302$ as the new hidden value.

These gate and candidate values are supplied activations, not results from a trained gate network. The new cell and hidden values are calculated from them. The example distinguishes keeping an internal value, writing a contribution, and exposing a value to the rest of the network.

This is the central state calculation of a common long short-term memory (LSTM) form with a forget gate:

$$c_t=f_t\odot c_{t-1}+i_t\odot g_t,\qquad h_t=o_t\odot\tanh(c_t).$$

The cell row $c_t$ is separate from the hidden row $h_t$. Both have the same number of coordinates. The symbol $\odot$ means multiply matching coordinates, not take a dot product or sum them. The forget gate $f_t$ controls the retained old cell contribution; a value near 1 forgets little. The input gate $i_t$ controls the new candidate $g_t$. The output gate $o_t$ controls the exposed hidden row. It does not directly choose an output token. The input and forget gates are independent: their values need not sum to one, which is why the cell can accumulate values outside the interval −1 to 1.

To specify where these changing multipliers come from, concatenate the current input and previous hidden row into $u_t=[x_t,h_{t-1}]$. If the input width is $m$ and the hidden width is $d$, this joined row has $m+d$ coordinates. A common version without direct cell-to-gate connections computes:

$$\begin{aligned} f_t&=\sigma(u_tA_f+b_f),& i_t&=\sigma(u_tA_i+b_i),\\ o_t&=\sigma(u_tA_o+b_o),& g_t&=\tanh(u_tA_g+b_g). \end{aligned}$$

Each $A$ has $m+d$ rows and $d$ columns, and each bias has $d$ coordinates. Each branch has its own learned parameters, reused over time. Sigmoid makes the gate coordinates into fractions; tanh allows a candidate to contribute positive or negative values. The scalar example simply enters this calculation after those activations have been supplied.

The separate cell route originated with LSTM; the adaptive forget gate was introduced in subsequent work on continual input streams. The equation above is the later forget-gate form, not an exact reproduction of every detail of the 1997 network. Hochreiter and Schmidhuber, 1997; Gers et al., 2000, §3

12.4.2 A single state path in GRU

We can also blend an old hidden value with a candidate without maintaining a separate cell. A gated recurrent unit (GRU) uses an update gate and a reset gate for these different decisions.

Example 12.4 — Reset the candidate’s view, not the retained branch. Supply old hidden value 0.8, update gate 0.9, and reset gate 0.25. Set the candidate’s current-input contribution to zero and its recurrent weight to 1. The reset gate leaves $0.25\times0.8=0.2$ available to form the candidate, so the candidate is $\tanh(0.2)\approx0.197375$. Blend 90% of the old value with 10% of this candidate: $0.9\times0.8+0.1\times0.197375\approx0.739738$.

The reset gate did not erase the separately retained 0.72 contribution. It changed the old information used to construct the candidate. Using the original Cho et al. convention, the equations are:

$$\begin{aligned} r_t&=\sigma(u_tA_r+b_r),\qquad z_t=\sigma(u_tA_z+b_z),\\ g_t&=\tanh(x_tW_g+(r_t\odot h_{t-1})U_g+b_g),\\ h_t&=z_t\odot h_{t-1}+(1-z_t)\odot g_t. \end{aligned}$$

Here $r_t$ is the reset gate and $z_t$ the update gate. The letter $z_t$ denotes a gate in these GRU equations, not Chapter 11’s output logits. The two $A$ matrices use the same joined-row dimensions as above; $W_g$ maps $m$ input coordinates to $d$ candidate coordinates, and $U_g$ maps $d$ reset hidden coordinates to $d$ candidate coordinates. All gates, candidates, and hidden rows have width $d$.

In this convention, larger update-gate values retain more of the old state. Some descriptions use the complementary convention, and some implementations place the reset operation differently. Read the equation before interpreting a gate value. This chapter and its code consistently use the form shown here. Cho et al., 2014, §2.3, equations (5)–(8)

Table 12.3 — Three answers to the same retention/update problem.

Mechanism	What crosses time?	How is old information retained?	How is new information admitted?
Simple RNN	Hidden state	Through recurrent weights and an activation	Input and old-state contributions are transformed together
LSTM, form above	Cell and hidden state	Forget gate scales a direct cell branch	Input gate scales a candidate; output gate exposes the cell through tanh
GRU, convention above	Hidden state	Update gate scales an old-state branch	Complementary update fraction admits a candidate; reset gate controls its access to old state

Gates add learned computations and parameters. GRU lacks the separate cell and output gate in this LSTM form, but that structural comparison is not a benchmark of speed or quality across hardware and tasks.

Gating also does not solve unlimited retention. Holding a forget gate at 0.9 for eight edges retains $0.9^8=0.43046721$ of the direct old-cell contribution; holding it at 0.99 for a hundred edges retains about 0.366032. A near-one gate offers a gentler path than repeated halving. It is still a product. Moreover, these calculations hold gates fixed to isolate the direct route. In the full network the gates themselves depend on earlier states, adding other gradient routes. Finite state size, overwrite, saturation, and failure to learn suitable gates remain possible.

12.5 A word’s representation can depend on its sentence

The changing hidden state offers another use besides producing the next token. Consider the same English token train in the train arrived and we train models. A static embedding lookup returns the same row for train in both sentences. A recurrent computation can combine that row with surrounding words to produce different contextual representations for the two occurrences. These vectors do not come with guaranteed labels such as “vehicle” and “verb”; their usefulness must be evaluated on a task.

ELMo, short for Embeddings from Language Models, made this a reusable pretraining approach. Its forward language model predicts the next token using the available left context. Its backward language model reads in reverse and predicts the preceding token using the right context. Each directional language-model objective keeps its own prediction target out of its input. When representing a complete supplied sentence for a downstream task, ELMo combines forward and backward states at each word, across stacked recurrent layers. “Bidirectional” here does not mean that a generator has access to words it has not generated yet.

Stacking means that one layer’s output at a position supplies the next layer’s input at that position; each recurrent layer also maintains its own state across positions. ELMo’s bottom token representation is computed from the word’s characters, before using surrounding sentence words. The recurrent layers add sentence context above it. Thus the static lookup in Figure 12.2 is a comparison baseline, not a claim that ELMo’s bottom representation is an ordinary word-table lookup. We need the distinction between the context-independent input and the contextual layers here, without implementing the character network. Peters et al., 2018, §3.1

Static-lookup baseline

1. Text piecesthe train arrived / we train models
2. A list of numberstrain → same lookup row

Lookup alone cannot distinguish these occurrences.

Pretraining

1. An operationForward: left context → next token
2. An operationBackward: right context → previous token
3. Stored model numbersLearn recurrent language-model weights

Two directional objectives; each keeps its own prediction target out of its input.

Task use

1. Stored model numbersFreeze pretrained weights
2. A list of numbersCombine directional states at each layer
3. Stored model numbersLearn layer weights, scale, task predictor

The complete supplied sentence changes the features; task training changes the mixture and predictor.

Figure 12.2 — The same token has one embedding row but can have different sentence features; task parameters train separately. Original teaching diagram and supplied or derived values, CC BY-SA 4.0; no paper figure reproduced.

Figure 12.2 separates learning reusable representations from using them. Pretraining learns the language-model parameters from text before training for a particular downstream task. In the original ELMo task recipe, these pretrained parameters are frozen. The task learns a weighted mixture of the token layer and recurrent layers, together with a scalar that scales the resulting vector; the layer weights are normalized by Softmax. The task’s own predictor is trained too. The mixture weights are task parameters, while the vectors being mixed change with the sentence. They are not attention weights over source positions. Peters et al., 2018, §3.1–3.3

This places ELMo between a context-independent word lookup and the broader use of deep pretrained contextual encoders. It is not BERT: ELMo’s recurrent language models process the two directions separately and combine features, whereas BERT uses a bidirectional Transformer with a masked-token pretraining objective. That distinction locates the architectures; the Transformer and masked training mechanisms come later. There is no requirement to train either model to understand this chapter. Devlin et al., 2019, §3.1

12.6 Summary

An RNN’s forward state path and its backward training signal pose related but distinct problems. BPTT follows the unfolded graph and adds contributions from repeated uses of each parameter. Products of local rates can make distant contributions tiny or very large. Gates provide more flexible retention and update paths, with extra computation and remaining limitations.

The same recurrent states can supply context-dependent features, as ELMo illustrates. So far, our main prediction task still reads an input stream and produces a local answer. Chapter 13 asks how one sequence can instead produce another sequence of a different length, and what must pass from the reader to the writer.

The optional CPU calculation is in code/part-iii/core.py; notebooks/part-iii/01-state-and-gradient.ipynb reproduces Tables 12.1–12.2 and both gate examples. Its backward loop is checked against the independent polynomial in Example 12.1 and a small finite-difference calculation. Neither that loop nor the distance table is a large-model training experiment.

12.7 Exercises

1. Transport a new signal. The final loss rate is 2. Four edges each have local rate 0.5. What rate reaches the earlier state? Repeat with factor 1.5. Explain why distance four means four multiplications.
2. Find a missing contribution. For Example 12.1 a program reports the weight gradient as −0.375 because it considered only the last multiplication. Identify the omitted contribution and calculate the correct parameter after a learning rate of 0.1.
3. Separate state from credit. A program passes a hidden state between two chunks but detaches its gradient at the boundary. A target in chunk two depends on a clue in chunk one. What information can still cross, and what learning route has been removed?
4. Operate the gates. Use old cell 0.8, forget gate 0.5, input gate 0.25, candidate −0.4, and output gate 1. Find the new cell and hidden value. Does output gate 0 instead erase the new cell? Interpret the exact 0/1 settings as limits.
5. Read a convention. In the GRU form above, set the update gate to 1 and the reset gate to 0. Does the hidden state become zero? Which branch does each gate affect?
6. Evaluate a representation claim. A proposal uses backward ELMo states computed from a complete reference translation to predict that same translation from scratch. Why is this not a valid generation evaluation? Describe a task where a complete input sentence is legitimately available.

12.8 Reference answers

1. The rates are $2(0.5)^4=0.125$ and $2(1.5)^4=10.125$. Distance counts dependency edges between the two states. Each edge transforms the small change once; the starting rate 2 is not another edge. These numbers concern sensitivity, not state magnitude.
2. The second reading contributes another −0.375; the first contributes zero because its previous state is zero. The total is −0.75, giving new weight 0.575. Summing uses follows parameter sharing; dividing by the number of readings would change this objective’s update.
3. The numerical value may still carry the clue forward. The second chunk’s loss cannot backpropagate through the detached boundary to credit the computations that formed that value in chunk one. A nonzero carried state does not prove full-sequence training.
4. The retained contribution is 0.4 and the new contribution is −0.1, so the cell is 0.3. The hidden value is $\tanh(0.3)\approx0.291313$. With output gate 0 it is zero, while the new cell remains 0.3. The output gate controls exposure, not this step’s cell update.
5. Update gate 1 retains the old hidden state unchanged and gives the candidate zero weight. Reset gate 0 removes the old-state contribution inside the candidate computation. It does not remove the separate retained branch. The answer would differ if a source defined the update gate with the complementary convention.
6. The backward features contain information from future reference tokens that a generator does not possess at that point. This leaks evaluation targets. Classifying the sentiment of an already supplied sentence, for example, legitimately allows both its left and right context. The availability boundary, not the word “bidirectional” alone, determines whether the use is valid.

12.9 References

• Bengio, Simard, and Frasconi, Learning Long-Term Dependencies with Gradient Descent Is Difficult (1994). Section 4 analyzes long-range learning under explicit dynamical assumptions.
• Hochreiter and Schmidhuber, Long Short-Term Memory (1997). Sections 3–4 connect gradient transport to the design of a cell path.
• Gers, Schmidhuber, and Cummins, Learning to Forget (2000). Section 3 introduces the adaptive forget gate for continual streams.
• Cho et al., Learning Phrase Representations using RNN Encoder–Decoder (2014). Section 2.3 gives the reset/update equations and the retained-old-state convention used here.
• Pascanu, Mikolov, and Bengio, On the Difficulty of Training Recurrent Neural Networks (2013). Section 2 writes gradients as sums of temporal contributions and products of local transformations.
• Zhang et al., Dive into Deep Learning, §9.7 and §10.1. These sections give a textbook derivation of BPTT and the modern LSTM gate calculation.
• Peters et al., Deep Contextualized Word Representations (2018). Sections 3.1–3.3 explain ELMo’s directional objectives, layer mixture, and use in downstream tasks.
• Devlin et al., BERT (2019). Section 3.1 and Appendix Figure 3 locate the difference from ELMo; the later chapters supply the missing Transformer mechanisms.