Chapter 13 Seq2Seq: Why Read-Then-Write Becomes a Bottleneck

13.1 Reading and writing need not take the same number of steps

The RNN in Chapter 11 reads a token and updates a state. Its color answer comes from a fixed pair of candidates. Translation asks for a different kind of output: a sequence whose length and word order need not match the input. Reading three source tokens might require four target predictions, including an instruction to stop. Which input should the writer receive at each of those predictions?

A simple arrangement is to finish reading first. One recurrent network, the encoder, transforms the source into a final vector. A second recurrent network, the decoder, uses that vector while writing the target one token at a time. This is a sequence-to-sequence (Seq2Seq) arrangement. Encoder and decoder have separate parameter sets; each set is shared across its own time steps. Recurrence permits variable lengths without a new parameter set for each position. Cho et al., 2014, §2.2

We will inspect two separate potential failures. The writer can encounter a prefix during generation that differs from the correct prefixes used for training. Independently, the reader can discard a distinction before the writer begins. A better treatment of one problem need not fix the other.

13.2 Read the source into a final vector

Example 13.1 — Compute every encoder state. Use a controlled translation, not an unrestricted Chinese-to-English task. The source tokens are exactly [红色, 钥匙, 。], meaning a red key followed by punctuation. The required target list is [red, key, ., EOS]. EOS is a special end-of-sequence token; it marks completion rather than a word to display. BOS, a beginning-of-sequence token, will start the decoder. It is an input marker, not one of the required output tokens.

To expose compression, choose a small encoder whose entire computation fits in a table. Let $a=\ln 2\approx0.693147$. Chapter 6 introduced the natural logarithm: $e^a=2$. We choose this constant so that the later attention calculation will have simple fractions. All vectors are dimensionless row vectors with two coordinates. These are new encoder weights, not Chapter 11’s halving weights.

Table 13.1 — The complete source vocabulary and encoder embeddings.

ID	Source token	Supplied embedding
0	红色 (red)	[a, 0]
1	蓝色 (blue)	[0, 0]
2	钥匙 (key)	[−a, a]
3	。 (full stop)	[0, −a]

The zero row for blue is a deliberate known-token embedding; it is not an unknown token or padding. Both input and recurrent matrices are the two-by-two identity, the hidden bias is [0,0], and the initial state is $h_0=[0,0]$. Thus each step adds the input to the previous state, then clips negative coordinates to zero:

$$W_x=W_h=\begin{bmatrix}1&0\\0&1\end{bmatrix},\quad b_h=[0,0],\qquad h_j=\operatorname{ReLU}(x_j+h_{j-1}).$$

The subscript $j$ counts source positions; we reserve $t$ for target positions. No output head is needed while computing this encoder’s states.

Table 13.2 — Encoding red and blue with the same weights.

Source reading	Red source: old state + input	Red source: new state	Blue source: new state
1: color	[0, 0] + [a, 0]	[a, 0]	[0, 0]
2: key	[a, 0] + [−a, a]	[0, a]	[0, a]
3: full stop	[0, a] + [0, −a]	[0, 0]	[0, 0]

For the blue source’s second reading, addition gives [-a,a]; ReLU turns that into [0,a]. This is the first point at which the two paths become exactly equal. The final full stop then yields [0,0] in both cases. All these states are calculated from the declared weights and inputs. They are not labels placed on an architectural drawing.

Call the final encoder state $c=h_3$ the context vector. In the simplest read-then-write design it is the only source representation handed to the decoder. For the diagram below, set the initial decoder state to $s_0=c$. A decoder update can use the previous decoder state, the embedding of the previous target token, and this same fixed context:

$$s_t=F(s_{t-1},E_y(y_{t-1}),c),\qquad p_t=\operatorname{softmax}(s_tW_o+b_o).$$

Here $s_t$ is a decoder hidden row; $E_y$ selects a target embedding row. $F$ stands for a recurrent update such as those in Chapters 11–12, with its own parameters. If the decoder width is $d$ and there are $K$ output candidates, $W_o$ has $d$ rows and $K$ columns and $b_o$ has $K$ entries. This is an architecture specification, not a claim that decoder weights have been supplied for a hand calculation. Other versions inject the source summary only through the initial decoder state. Both arrangements require information about the source to pass through the fixed summary. Sutskever et al., 2014, §2

Encode j = 1, 2, 3

1. A list of numbersRed → [a, 0]
2. A list of numbersKey → [0, a]
3. A list of numbersFull stop → c = [0, 0]

ReLU recurrence; a = ln(2). Earlier states are not handed to the baseline writer.

Decode t = 1, 2, 3, 4

1. A list of numberss₀ receives c
2. An operationBOS → red → key → . → EOS
3. Stored model numbersSeparate decoder parameters, shared in time

This row shows the required target path, not a measured prediction trace. Stop after predicting EOS.

Figure 13.1 — Three source readings feed one summary into four target predictions. Original teaching diagram and supplied or derived values, CC BY-SA 4.0; no paper figure reproduced.

Figure 13.1 separates the source clock from the target clock. It takes three updates to read this source and four predictions to produce its target including EOS. “Sequence to sequence” does not mean one output is paired with every source position. The source’s final punctuation is an ordinary token in our finite list; the list boundary tells this toy encoder when reading ends. Other data formats can include a separate source EOS.

13.3 Shift the target so the answer is not its own input

The target sentence is known in a training example. We need to use it as supervision without asking the model to predict a token it has just been handed at the same position. To predict red, supply BOS; to predict key, supply the previous correct token red. Continue this alignment through the end marker.

Table 13.3 — Training timeline: current input and required prediction.

Target step	Correct prefix before this step	Current decoder input	Token to predict
1	empty	BOS	red
2	red	red	key
3	red key	key	.
4	red key .	.	EOS

This is target shifting: prepend BOS to the input side, and line each preceding token up with the next target. EOS is a predicted target; it is not followed by another scored prediction here. There are four supervised positions, not three and not five.

Supplying the correct previous token during training is teacher forcing. “Teacher” refers to the dataset’s target, not a person intervening in each run. It sets the next input even if the model’s most probable prediction on the current step is wrong. With a recurrent decoder, knowing all these input tokens still does not let us compute every hidden state simultaneously: each state depends on its predecessor.

Example 13.2 — A complete conditional trace for inspection. To make the training/generation difference fully reproducible without fitting a translator, supply the following conditional probability table. The source context is held fixed. The decoder in this diagnostic looks up a row using the previous token. A general recurrent decoder can depend on the entire preceding prefix through its state; this small table deliberately simplifies that dependence. These probabilities are authored, not outputs derived from Example 13.1’s encoder or measurements from a trained decoder.

Table 13.4 — All reachable conditional rows in the diagnostic. Candidate order is [red, blue, key, door, ., EOS]; every row sums to one. Choose the largest entry, with the first candidate winning any tie.

Previous token	red	blue	key	door	.	EOS
BOS	0.4	0.5	0.1	0	0	0
red	0	0	0.8	0.1	0.1	0
blue	0	0	0.2	0.7	0.1	0
key	0	0	0	0	0.9	0.1
door	0	0	0	0	0.9	0.1
.	0	0	0	0	0	1

The exact zeros are part of a supplied lookup distribution; finite unmasked Softmax logits would give strictly positive probabilities. No EOS-input row is needed because generation stops when EOS is chosen. If EOS is not reached, the executable trace stops at a declared safety cap of eight predictions and reports that the cap, not EOS, ended the run.

Under teacher forcing, the four probabilities of the correct targets are 0.4, 0.8, 0.9, and 1. The first row prefers blue, yet the next training row still receives red. To score the complete reference sequence, multiply conditional probabilities along its correct prefix, just as in Chapter 4:

$$p(\text{red, key, ., EOS}\mid\text{source}) =0.4\times0.8\times0.9\times1=0.288.$$

With Chapter 6’s natural-log loss, the total is $-\ln(0.288)\approx1.244795$ nats. The mean is $1.2447947988461912/4\approx0.311199$ nats per target token. The divisor is four because EOS is included in this declared target sequence. A small mean on this one supplied sequence does not establish correct free generation: even its first most probable token is wrong.

13.4 Generation feeds its own choice into the next step

At generation time, the source is available but the correct target is not. Begin with BOS, choose a token, then use the chosen token as the next input. This is autoregressive generation: each choice becomes part of the prefix conditioning later choices. Chapter 4 used the same dependence when extending a sequence; here it operates inside a source-conditioned decoder.

Table 13.5 — Generation timeline using exactly Table 13.4.

Step	Current input	Largest probability	Chosen output	Next action
1	BOS	0.5	blue	Feed blue
2	blue	0.7	door	Feed door
3	door	0.9	.	Feed .
4	.	1	EOS	Stop

The reference is red key . EOS; the generated list is blue door . EOS. At step 2, the training trace used the red row, which favors key. Generation uses the blue row, which favors door. Both runs followed the same table. Their inputs diverged because only one run received the correct prefix.

Training inputs → targets

1. Text pieces1: BOS → red (p = 0.4)
2. Text pieces2: red → key (p = 0.8)
3. Text pieces3: key → . (p = 0.9)
4. Text pieces4: . → EOS (p = 1)

The correct red is supplied at step 2 even though step 1 favors blue.

Generation inputs → choices

1. Text pieces1: BOS → blue (p = 0.5)
2. Text pieces2: blue → door (p = 0.7)
3. Text pieces3: door → . (p = 0.9)
4. Text pieces4: . → EOS (p = 1)

The selected token becomes the next input; the period and EOS remain correct in this trace.

Figure 13.2 — The same conditional table sees different prefixes. Original teaching diagram and supplied or derived values, CC BY-SA 4.0; no paper figure reproduced.

Figure 13.2 makes the first mismatch explicit: the wrong prediction occurs at step 1; the resulting different next input occurs at step 2. Training on reference prefixes while running on the model’s own prefixes creates a distribution difference often called exposure bias. The name identifies a potential difficulty with the contexts the decoder encounters; it does not by itself explain why the initial prediction was wrong. S. Bengio et al., 2015, §1–2

This trace demonstrates one way an early error changes later input and output. It does not measure how frequently that happens in a real translator, prove that errors always compound, or show that teacher forcing is ineffective. The period and EOS recover here. If the blue row instead strongly favored key, the initial color error could remain without causing the noun error. Changing a training procedure needs evaluation on held-out generation cases, not a conclusion drawn from this one authored table.

13.5 A writer cannot recover a distinction its only source representation lost

Teacher forcing and compression are separate issues. Even if we supply the correct target prefix at every step, the final vectors in Table 13.2 remain identical for the red and blue sources. At the first target step the decoder has the same BOS and the same context, so a fixed deterministic decoder must produce the same distribution. It cannot reliably give different first colors for those two inputs. Sampling can produce different strings, but without a source-dependent distribution it does not restore the missing distinction.

This example is deliberately severe: its small encoder loses the color after just two readings. It proves a collision for these weights, not that every Seq2Seq encoder loses color or fails on every long sentence. Better weights, more coordinates, and gating can preserve more useful distinctions.

Still, the fixed-vector arrangement places a general burden on the encoder: all source information the decoder will need later must be made accessible through one vector before the first output is written. A longer source can contain more independently relevant details. With a fixed number of finite-precision coordinates, there are finitely many stored patterns. There cannot be a different pattern for every arbitrarily long possible input. This counting observation does not predict where a trained model will fail; a task may need only a small summary rather than the whole input. Nor does vector dimension alone prove an impossibility for ideal real numbers of unbounded precision. The practical issue is representing and learning to retrieve the distinctions the task requires.

An LSTM cell can improve the path along which information travels without making one summary an unlimited store. Conversely, changing from reference prefixes to generated prefixes during an experiment does not change the encoder’s two equal vectors. The diagnosis should identify which information path failed before proposing a remedy.

The reader had useful intermediate states before the final compression. Could the writer retain access to them and obtain a different combination for each output position? That proposal changes the source-to-output route rather than merely increasing the fixed vector’s size.

13.6 Analyze the trace as a reproducible practice task

data/part-iii/sequence-v1.json contains the source vocabulary, encoder weights, exact coefficients of ln(2), full conditional table, candidate order, target shift, and stop rule. code/part-iii/run.py computes both encoder paths and follows both decoder timelines. The second notebook, notebooks/part-iii/02-seq2seq-attention.ipynb, continues this trace into Chapter 14. It requires only the Python primer; reading the supplied tables remains a complete practice path.

The practical acceptance criterion is not “a translator was trained.” It is to reproduce or fully analyze a sequence failure: locate the first wrong prediction, the first changed input, and the encoder’s first state collision; then explain why those are different events. Their answers are already checkable from Tables 13.2–13.5. The saved program run verifies this diagnostic behavior on a CPU. No training time, translation quality, or generalization measurement is claimed.

13.7 Summary

An encoder can read one sequence into a context that a decoder uses to write another. Target shifting aligns the previous token with the next supervised target, including the beginning and end boundaries. Teacher forcing supplies correct previous tokens; autoregressive generation supplies the model’s chosen tokens. The resulting prefix difference can affect subsequent predictions.

A fixed source vector raises a second issue: different sources can become indistinguishable before decoding begins. Chapter 14 keeps the recurrent encoder and decoder, but lets each target step read from the collection of encoder states. We will calculate exactly what this extra access returns.

13.8 Exercises

1. Shift a different target. For target [blue, door, ., EOS], list all decoder input–target pairs under teacher forcing. Does BOS contribute a target loss? Does EOS?
2. Separate prediction and feedback. In Table 13.4, why does the first wrong prediction not cause the training trace’s second input to be blue? Locate the first input divergence during generation.
3. Change the cascade. Replace only the blue row by [0,0,0.8,0.1,0.1,0]. Compute the full greedy output. What conclusion about error propagation does this support, and what conclusion does it not support?
4. Find the collision. Show the blue encoder’s preactivation and post-ReLU state at the key reading. At which reading do red and blue become identical? Would perfect teacher forcing distinguish their first predicted colors?
5. Interpret a score. Compute the mean target loss if the existing reference is scored without EOS. Explain which parts of the calculation change, even though the generation rule does not.

13.9 Reference answers

1. The pairs are BOS→blue, blue→door, door→period, period→EOS. BOS is only an input and has no target loss; EOS is a target and does. There are four scored predictions.
2. Teacher forcing chooses the next input from the reference, not from the model’s maximum-probability output. Thus the training input at step 2 is red. Generation feeds its step-1 choice blue into step 2, which is the first input divergence; the first prediction error was one step earlier.
3. The new output is blue key . EOS. The color remains wrong but the noun is correct. An early error need not cause every later output to be wrong. One altered diagnostic row cannot establish that exposure bias is harmless in general.
4. Blue supplies a zero first embedding. At key, its preactivation is [-a,a], and ReLU produces [0,a], identical to the red path at reading 2. The final contexts are also identical. At the first target step both have BOS and the same context; teacher forcing provides no earlier color token to distinguish them.
5. Removing EOS removes a loss of $-\ln1=0$, so the numerical total stays 1.2447947988461912; the definition of the total has one fewer term. Dividing by three instead of four gives about 0.414932 nats per target token. Scores using different event definitions cannot be compared as if they measured the same average; no generation rule has changed.

13.10 References

• Cho et al., Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014). Section 2.2 defines source-conditioned decoding and different sequence lengths; its experiment is distinct from our authored trace.
• Sutskever, Vinyals, and Le, Sequence to Sequence Learning with Neural Networks, arXiv v3 (2014). Section 2 describes the final source representation, separate LSTMs, target factorization, and EOS. Its successful long-sentence results also caution against treating the toy collision as a universal failure.
• S. Bengio et al., Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks, arXiv v3 (2015). Sections 1–2 describe the training/generation prefix difference; this chapter does not recommend a training repair or transfer the paper’s measurements.
• Bahdanau, Cho, and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate (2015). Sections 2–3 identify the fixed-summary burden and motivate access to multiple encoder annotations, the question pursued next.