Chapter 11 RNNs: Carrying the Past into the Present

11.1 What can survive after a clue leaves the window?

Read blue key is and answer with the key’s color. A model receiving only the final two tokens sees key is. Replace blue with red, and its input is unchanged although the required answer changes. Chapter 10 showed why changing weights cannot recover a distinction that never reaches the computation. Enlarging the window postpones that boundary, but we would have to choose its size in advance.

Suppose instead that reading blue produces a small numerical record. Reading key uses both its own input and that record to produce a new record. Reading is repeats the operation. The current input still has a fixed size, but an earlier input now has a route to the current prediction. Does having that route suffice to answer correctly?

We will test the proposal on the same red/blue task, first with three tokens and then with the longer pair from Chapter 10. The criterion is the color of the first token, chosen from the fixed candidate order [red, blue]. We always select the candidate with the higher output probability; ties go to the first candidate. These are diagnostic examples with supplied weights, not a trained language model or a claim about typical RNN accuracy.

The arithmetic uses Chapter 5’s row vectors, Chapter 6’s Softmax, and Chapter 7’s ReLU and training distinction. The required operations are recalled below. Code is optional; its prerequisite is the Python primer.

11.2 Use the same operation at three different times

Example 11.1 — Carry a red clue through three readings. The input is exactly red key is. Each space-separated word is one token in this example. The vocabulary, IDs, and supplied embedding rows are:

Table 11.1 — Every input row in the diagnostic model.

Token	ID	Input vector
red	0	[1, 0]
blue	1	[0, 1]
filler	2	[0, 0]
key	3	[0, 0]
is	4	[0, 0]

IDs select rows; they are not quantities to multiply. Zero rows here are deliberate choices for known tokens, not missing words or padding. The two coordinates make the color example easy to inspect; ordinary learned coordinates do not come with these meanings built in.

Start the retained record at [0, 0]. At each reading, halve its two old coordinates, add the current input coordinate by coordinate, and replace any negative result by zero. This last operation is ReLU: it keeps positive values and clips negative ones. For example, applying it to [-1, 0.5] produces [0, 0.5]. Our three steps stay nonnegative, so clipping does not change their results.

Before writing an equation, follow the timetable. The parameter settings do not depend on the time row.

Table 11.2 — One set of parameters, three uses.

Reading	Current input	Old record	Parameters used	New record
1: red	[1, 0]	[0, 0]	Keep input; multiply old record by 0.5	[1, 0]
2: key	[0, 0]	[1, 0]	The same settings	[0.5, 0]
3: is	[0, 0]	[0.5, 0]	The same settings	[0.25, 0]

At reading 2, key contributes zero, but the calculation is not starting from zero: half of [1, 0] arrives from reading 1. At reading 3, the available past is already [0.5, 0]. We halve that value, obtaining [0.25, 0], rather than returning to the original embedding of red.

The retained numerical record is the hidden state. “Hidden” means an internal activation rather than an observed answer; it does not mean inaccessible to inspection. A network that computes the next state from the current input and previous state is a recurrent neural network, or RNN. The recurrence is this repeated state update, not the reappearance of an English word in the sentence. Elman’s simple recurrent network provides an early formulation of feeding earlier hidden activations into later computation. Our small weights are a separate teaching construction. Elman, 1990

We can now name the quantities. Let $t$ count input readings, $x_t$ be the current input row, and $h_t$ be the hidden row after that reading. Both have two coordinates here. The initial row is $h_0=[0,0]$. All weights and biases are supplied below:

$$W_x=W_y=\begin{bmatrix}1&0\\0&1\end{bmatrix},\qquad W_h=\begin{bmatrix}0.5&0\\0&0.5\end{bmatrix},\qquad b_h=[0,0],\quad b_y=[0.125,0].$$

Each matrix has two rows and two columns. Multiplying a row by the identity matrix leaves it unchanged; multiplying by $W_h$ halves each coordinate without mixing the two. The hidden bias $b_h$ adds nothing. The output bias $b_y$ supplies a small preference for red, so the test will require enough blue evidence to overcome it. With the five embedding rows in Table 11.1, these are all the model’s numerical parameters.

$$h_t=\operatorname{ReLU}(x_tW_x+h_{t-1}W_h+b_h),\qquad z_t=h_tW_y+b_y,\qquad p_t=\operatorname{softmax}(z_t).$$

The first expression writes the operation in Table 11.2. The row $z_t$ then contains two output scores, or logits, in [red, blue] order. The row $p_t$ contains their probabilities. A hidden coordinate and a probability are different objects: hidden coordinates need not add to one, whereas the two probabilities must. The quantities in this example are dimensionless numbers, not seconds, counts, or amounts of money; $t$ is a position, not elapsed clock time.

For the final reading, substitution gives $h_3=[0.25,0]$ and $z_3=[0.375,0]$. Chapter 6’s rule exponentiates each score and divides by the sum for the two output candidates:

$$p_3(\text{red})=\frac{e^{0.375}}{e^{0.375}+e^0} \approx\frac{1.454991}{1.454991+1}\approx0.592667.$$

The denominator is not the number of input tokens or hidden coordinates. It is the total of the positive weights assigned to the output candidates. Blue receives the remaining probability, about 0.407333, so this example answers red.

Table 11.3 — The complete three-step output log.

Reading	New hidden state	Logits: red, blue	Probabilities: red, blue
1	[1, 0]	[1.125, 0]	[0.754915, 0.245085]
2	[0.5, 0]	[0.625, 0]	[0.651355, 0.348645]
3	[0.25, 0]	[0.375, 0]	[0.592667, 0.407333]

Tables 11.1–11.3 jointly expose every input, old state, new state, and output. Intermediate color predictions are shown for inspection; this task is scored only after is. A next-token language model can use the same recurrent pattern with a different output vocabulary and a prediction target at each reading.

Reading 1

1. Text piecesInput: red
2. A list of numbersOld h: [0, 0]
3. Stored model numbersSame Wₓ, Wₕ, bₕ
4. A list of numbersNew h: [1, 0]

Input + old state → update → new state → next reading.

Reading 2

1. Text piecesInput: key
2. A list of numbersOld h: [1, 0]
3. Stored model numbersSame Wₓ, Wₕ, bₕ
4. A list of numbersNew h: [0.5, 0]

Input + old state → update → new state → next reading.

Reading 3

1. Text piecesInput: is
2. A list of numbersOld h: [0.5, 0]
3. Stored model numbersSame Wₓ, Wₕ, bₕ
4. A list of numbersNew h: [0.25, 0]

Input + old state → update → new state → next reading.

Figure 11.1 — Three uses of one recurrent update; the numerical state crosses time. Original computed teaching example, CC BY-SA 4.0. Double borders mark shared parameters.

Figure 11.1 unfolds time so that the input to each update is visible. The repeated parameter boxes refer to the same stored matrices, not three models that can learn different values. The arrows carrying states are computation dependencies: the third update needs the second result.

11.3 Test the new route by changing the clue and its distance

Example 11.2 — Blue tests whether the clue matters. Change only the first token to blue. The states become [0,1], [0,0.5], and [0,0.25]. The final logits are [0.125,0.25]; blue wins with probability 0.531209. The rule did not change. A different input propagated through the state and changed the chosen answer.

For a fair window comparison, use Table 11.1’s embeddings and the same output head. Concatenate the final two embedding rows, sum their corresponding coordinates, apply ReLU, and then add [0.125,0]. This is a fully specified two-token feedforward baseline: its four-to-two weight rows are [1,0], [0,1], [1,0], [0,1], with zero hidden bias. It is not the trained NNLM from Chapter 10. Because the two final words are always key is, this baseline always obtains [0,0] before the output head and assigns red probability 0.531209.

The short RNN comparison passes both colors. The short window comparison passes only red. Now restore Chapter 10’s long input pair without changing the criterion or either model’s parameters:

red  filler filler filler filler filler filler key is
blue filler filler filler filler filler filler key is

There are nine tokens in each row. After the first reading, eight updates multiply the retained clue by one half. The surviving coordinate is $0.5^8=1/256=0.00390625$.

Table 11.4 — The same long task exposes two different failures.

Input clue	Two-token window logits	RNN final state	RNN logits	RNN probability of red	RNN choice
red	[0.125, 0]	[1/256, 0]	[0.12890625, 0]	0.532182	red: correct
blue	[0.125, 0]	[0, 1/256]	[0.125, 0.00390625]	0.530236	red: incorrect

Both methods now pass one of these two deliberately chosen cases. Their reasons differ. The fixed window gives exactly the same input and distribution for the pair. The RNN has different states and distributions, but its tiny blue contribution is insufficient to overcome the red bias. The information route exists; this setting does not preserve a useful margin for the final decision. Calling both failures “no memory” would conceal the distinction needed to repair them.

The RNN’s blue coordinate has not rounded to zero. Even with exact fractions, it loses the comparison against 1/8. Thus this failure does not require floating-point underflow. Changing the retention weight or output bias could change the outcome. It would not establish that all distant clues in an ordinary language task can be preserved.

11.4 State changes during a run; parameters change during learning

An RNN can process another token without adding another set of parameters. This parameter sharing lets one update rule operate on varying sequence lengths. It does not guarantee equal performance at every length, as Table 11.4 demonstrates.

During an ordinary model run (inference), the supplied matrices stay fixed while each input produces a new hidden state. At the start of each independent red/blue case we reset the state to zero. Otherwise the second case could inherit numbers from the first, and we would no longer be testing the stated inputs independently. An application may intentionally carry state across chunks of one sequence, but that boundary must be explicit.

Training adds a different operation. The network first computes states and predictions, then compares the relevant predictions with targets to obtain a loss. Gradients measure how parameter changes would affect that loss; an optimizer then updates the parameters. In the full-sequence training procedure considered here, the shared weights remain fixed throughout that forward pass. After an optimizer update, a new pass computes new states from the new weights. State updating alone is not learning.

Table 11.5 — What changes, and what causes the change?

Object	Ordinary forward run	Training step
Input tokens	Supplied in sequence	Supplied from the training example
Hidden state	Recomputed at every reading	Recomputed in the forward pass; activations are retained or recomputed for gradients
Shared parameters	Fixed	Changed by the optimizer after gradient computation
Correct color target	Not available to choose an answer	Used to evaluate the loss

There is still a scheduling cost. We cannot calculate $h_3$ by starting three independent state updates simultaneously, because $h_3$ depends on $h_2$, which depends on $h_1$. Matrix operations within a step and separate sequences in a batch may run in parallel. This does not remove the dependency between readings of one sequence. No hardware timing is claimed by this argument.

The hidden state is also not a stored transcript, an application’s long-term memory, or an Agent State. The last of these may contain explicit records such as an order ID, a tool result, or whether cancellation was approved. A hidden vector is calculated by the network’s weights; it neither supplies a missing order lookup nor grants permission to cancel an order. Likewise, a recurrence cannot insert the updated 750 CNY policy if the application supplied only the old 600 CNY document. Chapter 1’s system boundaries still apply.

11.5 Reproduce the failure without training a large model

The shared fixture data/part-iii/sequence-v1.json fixes the embeddings, weights, candidate order, and translation data used later. The long prefixes are read from the existing Chapter 10 fixture, whose path and hash are recorded; they have not been replaced with easier examples. expected.json preserves independent fraction-based expectations. Numbers above are derived from these teaching rules; the accompanying run.json is the actual program’s reproduction, not a training log.

After the Python primer, run the following from the book repository root. It uses Python’s standard library on a CPU, performs no downloads, and prints the complete state and failure logs.

python3 code/part-iii/run.py

The first notebook, notebooks/part-iii/01-state-and-gradient.ipynb, carries the same calculation into Chapter 12. The learning milestone is to identify the first update after which blue no longer wins, and explain the reason using its state and the red bias. Reading Tables 11.2–11.4 and answering Exercise 1 achieves the core milestone without running code.

11.6 Summary

A fixed window limits which tokens enter a prediction. Recurrence adds a computed state through which earlier tokens can influence later predictions. The three unfolded steps reuse one parameter set, while their inputs and states differ. This creates a longer information route and a sequential dependency; it does not promise reliable retention.

Our distant blue clue still reaches the output, but with too little influence to win. Training might change the weights responsible. Chapter 12 asks how an error at the end sends a useful training signal back through the very same sequence of updates.

11.7 Exercises

1. Find the boundary. Insert filler tokens between blue and key is. With the original weights and tie rule, what is the smallest number of fillers that makes the RNN choose red? Show the final state and both logits.
2. Change one parameter. Set the diagonal entries of the recurrent matrix to 1, keeping all other values fixed. Compute the final state and prediction for the long blue case. What new problem could occur for a stream containing many blue tokens?
3. Diagnose a log. A programmer reports that processing red, key, and is changed the model because the hidden row changed from [1,0] to [0.25,0]. What evidence would establish an actual parameter update? Why would resetting the state before every token break Example 11.1?
4. Choose evidence. A system answers both long colors correctly. Does this establish that it used an RNN, that its weights changed, or that it can retrieve the current travel policy? Give a controlled observation needed to investigate one of these claims.
5. Trace responsibility. An order assistant’s hidden vector changes after generating approved. Its application record still says cancellation approval is absent. Which record governs the permitted action, and why does the changed vector not replace it?

11.8 Reference answers

1. With no filler, the blue coordinate is 1/4. One filler adds one halving, leaving 1/8. The logits become [1/8,1/8], so the declared tie rule chooses red. One filler is sufficient; two would make blue strictly smaller. Counting the transitions after the clue, rather than all tokens, avoids an off-by-one error.
2. Every zero input now leaves the previous state unchanged. The long blue case ends at [0,1], with logits [0.125,1], so blue wins. Repeated blue inputs keep adding one to that coordinate; there is no selective reset or overwrite. Preserving this isolated clue is not a general solution to deciding which information should remain.
3. Compare saved parameter values before and after the alleged update, together with the optimizer operation responsible. Different activations under fixed weights are expected. Resetting before every token removes the previous-state contribution, so the final is produces [0,0] and the output head prefers red regardless of the clue.
4. None follows from two correct strings. For example, to investigate learning, hold the input and selection settings fixed and compare parameter snapshots and update records. A correct color response cannot establish that a different application supplied a current policy document.
5. The application’s explicit approval record and enforcement rules govern the action. A generated token and its hidden activations are outputs of a numerical computation. They are not an approval event or evidence of an authorized cancellation.

11.9 References

• Elman, Finding Structure in Time (1990). Pages 181–185 explain how previous hidden activations supply temporal context; the book’s ReLU weights are original.
• Cho et al., Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014). Section 2.1 states the recurrent update and output-distribution pattern; later sections motivate the next chapters.
• Zhang et al., Dive into Deep Learning, §9.7. The introduction connects an unfolded time diagram to shared parameters and prepares the training question of Chapter 12.