Chapter 18 Transformer Families: BERT, GPT, and T5

We have a block that can exchange information among positions and transform their representations. But “use a Transformer” does not yet specify a learning problem. Should a missing word use text on both sides? Should a continuation see only its prefix? Should a new answer read an entire separate document while being generated one token at a time?

These are different information arrangements. This chapter uses the same small sentence to compare three influential combinations: BERT, the original GPT approach, and T5. The aim is to locate their structure and objectives, not reproduce their training or rank their quality. Their papers describe particular models and experiments; a later model or a chat product need not preserve every original choice.

18.1 Start with one sentence and three questions

Example 18.1 — What is the target in red key .? For this visibility exercise, each word and the full stop is one authored token. These are not the actual token IDs of BERT, GPT, or T5. We omit source/classification/separator boundary tokens to keep the grids small, but show decoder BOS and terminal EOS where needed. A real implementation must follow its own tokenizer and boundary conventions.

First ask: “Which token belongs in the first slot, given the remaining sentence?” We hide red and input [MASK] key .. The output at the hidden slot predicts red. The query can use both key and the full stop, since both are available evidence. It can also read the representation of [MASK], which marks the missing location without supplying the answer.

Second ask: “What comes next, given only what has already appeared?” Supply [BOS,red,key,.] during training and score [red,key,.,EOS] at the corresponding positions. All known input tokens can be processed in one forward pass with a causal triangle. The output above red predicts key, so it may read BOS and red, but not the following key input.

Third ask: “Recover a removed span, using the remaining source.” Replace the two-token span red key with a special marker <S0>, giving source [<S0>,.]. The desired target is [<S0>,red,key,<S1>,EOS]. <S0> identifies the removed span, and the final sentinel <S1> marks the end of the span-recovery sequence before EOS. These sentinels are reserved tokens, not prose instructions or randomly discovered words. If multiple spans were removed, separate sentinels would identify their boundaries.

Table 18.1 — Three constructed input/target contracts for the same original sentence. “Scored” means included in that objective’s loss, not merely computed by the network.

Pattern	Network input	Scored targets	Available evidence for predicting red
BERT-style MLM	[MASK] key .	red at input slot 1	All three corrupted input slots
GPT-style causal LM	BOS red key .	red key . EOS, shifted	BOS only at the first prediction
T5-style span denoising	Source <S0> .; decoder input BOS <S0> red key <S1>	Decoder targets <S0> red key <S1> EOS	The whole corrupted source and decoder prefix BOS <S0>

The same word red is therefore predicted under different conditions. A lower loss on one row of this table cannot be directly read as better language skill: the evidence and the number of scored targets differ.

18.2 A bidirectional reader and selected missing-token targets

BERT uses an encoder-only stack: self-attention can combine both sides of a real input position. For our corrupted three-slot input, the visibility grid has three query rows and three key columns. Every cell is allowed. Its output representations remain aligned with the three input positions, and the selected first position is trained to predict the missing token. This is masked language modeling, or MLM.

There are two different meanings of “mask” here. Replacing red with [MASK] changes the input token. The visibility mask controls which other positions a query can use. BERT’s example hides the answer token while allowing bidirectional attention; it does not hide the [MASK] position from all queries. Confusing the two would describe a different training problem.

Original BERT selects 15% of token positions for the MLM objective; among selected positions, 80% are replaced by [MASK], 10% by a random token, and 10% kept unchanged. Example 18.1 shows one selected position taking the [MASK] branch; it does not simulate those corpus-wide proportions. Original BERT also has a next-sentence prediction objective. Neither that additional objective nor the exact corruption percentages define all encoder-only models. BERT, Section 3.1.

For a task that labels an already available sentence, seeing both sides is compatible with the task. Using this unmodified bidirectional arrangement to score a next-token predictor on an intact sentence would be incompatible: it could see the target. An encoder can participate in systems that generate or fill text, but ordinary repeated next-token generation is not the same operation as one BERT MLM pass.

18.3 A continuation model with a causal prefix

The original GPT approach uses a decoder-only Transformer language model and a causal language-modeling objective. Here “decoder-only” means a stack that generates from a prefix without a separate encoder and its cross-attention path. It is not the entire original encoder-decoder Transformer with its encoder left dangling.

For the four-slot input in Table 18.1, allowed key sets are [1], [1,2], [1,2,3], and [1,2,3,4]. There are ten allowed query-key pairs, not sixteen. The last output predicts EOS after reading the full sentence. At generation time, future inputs are not provided at all; the program chooses one token, appends it, and runs the model again. Teacher forcing permits parallel scoring during training, while generated tokens still depend on previous choices.

GPT’s original study then adapts pretrained parameters to supervised downstream tasks. That is a training-stage distinction, not a new visibility category. A causal language model can also produce a classification label after reading a complete input prefix. It is not restricted to entertaining continuations, and being decoder-only does not imply that a model has undergone conversational instruction training. Original GPT, Sections 3.1–3.3.

18.4 A source reader paired with a causal writer

T5 uses an encoder-decoder arrangement. Its encoder reads the entire supplied source. Its decoder combines two different attention relationships: causal self-attention over the decoder input, and cross-attention to all valid encoder output positions. The decoder’s query supplies Q for that cross-attention, while the encoded source supplies K and V. “May see the whole source” does not mean “may see the future target.”

In our example, the source has two slots, so encoder self-attention permits all four source pairs. The five decoder input slots allow 1+2+3+4+5=15 causal pairs. Decoder-to-source attention permits five times two, or ten, pairs. These are separate matrices, with shapes [2,2], [5,5], and [5,2] per head. A cross-attention matrix need not be square because source and target lengths can differ.

BERT-style: selected missing slot

1. Text piecesInput: [MASK] | key | .
2. An operationEach query → keys 1,2,3 (9 pairs)
3. Text piecesScore slot 1 → red

All input slots are visible, but red has been replaced.

GPT-style: next token at every step

1. Text piecesInput: BOS | red | key | .
2. An operationAllowed keys: 1; 1,2; 1,2,3; 1,2,3,4
3. Text piecesTargets: red | key | . | EOS

The first red prediction has only BOS as evidence (10 causal pairs in total).

T5-style: recover a removed span

1. Text piecesSource: <S0> | .
2. An operationEncoder 2×2: all 4 pairs; decoder 5×5: 15 causal pairs
3. An operationCross-attention 5×2: all 10 source pairs
4. Text piecesDecoder prefix BOS | <S0> → red

The source is fully readable; future target tokens remain hidden.

Figure 18.1 — The same target has three different evidence sets. Original teaching diagram, CC BY-SA 4.0. Values and rules are declared in the chapter and shared data.

Figure 18.1 follows the evidence used to predict red. For T5, when <S0> is already in the decoder prefix, its output can use the whole corrupted source and that prefix. It cannot read the upcoming target token red from the decoder input. The source must not accidentally contain the removed span either. If we fed the intact red key . as source for this corruption objective, we would have changed the task into copying the answer from an available source.

This form of denoising constructs a training target by corrupting a complete original record and learning to recover the missing content. T5’s span-corruption target contains the removed spans and sentinels, not necessarily the entire original sentence in its original positions. During downstream use, tasks can be represented as source text and target text. The network’s internal normalizations and other details need not match Chapter 17’s toy block. T5, Section 3.1.4 and Figure 2; Section 3.2.

18.5 Keep four classification axes apart

Table 18.2 — Four questions that a model name alone cannot answer.

Axis	What to inspect	Possible answer
Architecture	Which stacks and attention connections exist?	Encoder-only, decoder-only, encoder-decoder
Training objective	Which input is supplied, what target is scored, with what visibility?	MLM, causal next-token prediction, span denoising
Training stage	What data and purpose govern this parameter update stage?	Broad pretraining, task fine-tuning, instruction-oriented post-training
Product or interface	How does a user supply input and receive output?	Chat interface, classifier endpoint, text-generation application

The historical combinations are helpful landmarks, not an exhaustive set of allowed pairings. For example, an encoder-decoder model can learn continuation by treating a prefix as source and its suffix as target. A decoder-only model can be trained on corrupted-source/target sequences with a specified visibility and loss policy. T5’s study explicitly compares structure and objective as separate experimental choices; its results apply to its data and evaluation setup, not a universal ranking of families.

Likewise, a chat interface does not tell us the underlying architecture, data, training stage, or whether a response came from an external tool. “LLM” is not synonymous with either a chat window or one architectural branch. None of these categories supplies authority to query or cancel a real order. In the book’s order case, the application must still determine which operation is permitted and what evidence counts as an actual result.

18.6 Choose for a specified task, then test the choice

Example 18.2 — Three versions of the same key task. Suppose we have labeled records and enough budget for a small controlled comparison. A choice should say what inputs are available and how success is measured. Table 18.3 gives feasible starting designs and a reason to reject a tempting assumption.

Table 18.3 — A choice is conditional on the task.

Task and criterion	Feasible starting design	Reason to reject an assumption
Given the whole red key . or blue key ., output one color label; measure exact label accuracy on held-out records	Encoder representations plus a classifier; causal LM’s final representation or a generated label are alternatives	Do not call bidirectional reading leakage when the whole sentence is legitimately supplied
Given red key, continue with the next token; score held-out next-token loss under the same prefix	Causal LM; encoder-decoder continuation with the same source/target contract is also feasible	Do not evaluate an unmodified bidirectional intact-input pass as though it saw only the prefix
Given a noisy key instruction, output a corrected sequence; measure specified edits and preserved content	Encoder-decoder source-to-target training; a causal source-prefix/answer format is an alternative	An encoder-decoder shape does not itself teach correction; suitable pairs and evaluation are needed

For a fair empirical comparison, freeze records and splits, specify equivalent task evidence, control or report compute and model size, define target formatting, and evaluate the same task metric. If tokenizers differ, raw “nats per token” denominators differ too; they are not automatically comparable. We have made architectural decisions here, not measured their relative performance.

The fictional Beijing limit illustrates the same boundary. Replacing 600 with 750 in an available updated policy document may make the new value accessible to any suitably arranged model. A family name does not establish that the correct version was supplied, that the model followed it, or that its parameters were updated. Architecture describes paths; evidence must show what actually traveled through them.

18.7 What the family map lets us ask

BERT-style missing-token prediction can use both sides of a corrupted input. GPT-style continuation restricts each output to its prefix. T5-style span recovery separates a full source reader from a causal target writer. These descriptions identify information and targets more precisely than a family or product label alone.

For our Mini GPT we will choose a decoder-only causal objective. We still need to convert actual text into stable IDs, preserve boundaries, and align inputs with targets. Chapter 19 addresses that interface before Chapter 20 assembles the model.

18.8 Exercises

1. Draw or list the allowed keys for each row of the BERT-style three-slot example and the GPT-style four-slot example. Explain why all BERT pairs can be allowed without revealing the missing red token in the stated corruption branch.
2. In the T5-style example, the decoder is about to predict key. State its available decoder prefix, its available source, and the number of visible decoder and source positions for that query.
3. A model has an encoder and a causal decoder but is trained to continue a document prefix. Must its architecture be called GPT? Identify architecture and objective separately.
4. A vendor supplies a chat interface and says its model received instruction fine-tuning. Which of Table 18.2’s axes have been described, and which remain unspecified?
5. A comparison gives an MLM model lower loss than a causal model on the same original sentences. What must you inspect before interpreting this as better performance? Suggest a common downstream test.

18.9 Reference answers

1. Each BERT query allows keys [1,2,3]; GPT allows [1], [1,2], [1,2,3], [1,2,3,4]. The BERT input contains [MASK], not red, at the selected position. Its objective intentionally uses the remaining right context. Visibility is judged against the declared task, not against a rule that all language models must be causal.
2. The decoder input prefix is [BOS,<S0>,red], three positions. The source is [<S0>,.], two positions. This query can read all three prefix positions through self-attention and both source positions through cross-attention. The upcoming key target is in neither allowed set.
3. Architecture: encoder-decoder. Objective: source-conditioned continuation of a prefix. A particular causal objective does not convert two network stacks into a decoder-only architecture or assign a historical model’s product name.
4. The interface and one training stage have been described. The stacks/connections and the complete objectives remain unspecified, as do the data, tested capabilities and tools. Instruction fine-tuning does not by itself identify the architecture.
5. Inspect corruption, available left/right context, selected target positions, tokenizer, reduction denominator and split. MLM may receive right context that the causal model cannot use and score fewer events. A shared held-out color-classification task can supply the whole sentence to both, define the same exact-label criterion, and report training resources. The result would be about that task, not all language abilities.

18.10 References

• Devlin et al. (2019), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Sections 3.1–3.2 and Figure 3 support the reader/MLM setup, original corruption choices, and distinct adaptation stage.
• Radford et al. (2018), Improving Language Understanding by Generative Pre-Training. Section 3 explains a causal Transformer language model followed by supervised adaptation; it is not documentation of a current chat product.
• Raffel et al. (2020), Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Figure 2 and Sections 3.1.4–3.2 explain sentinels, span targets, and independently varied architectures/objectives. Our key sentence and decision table are original teaching constructions.