Chapter 20 Building a Mini GPT from Scratch

We can calculate an attention head, restrict its information, assemble a block and specify a tokenizer. The remaining challenge is to make all those contracts hold at once. A program might return a plausible loss even when its labels are not shifted, its padding affects the denominator, or its vocabulary rows have changed meaning.

MG-1 is the first complete neural Mini GPT milestone: raw text reaches logits and loss through a small, inspectable decoder-only model. It builds on Chapter 7’s learning loop and Chapters 15–19’s mechanisms. This chapter completes the forward route; Chapter 21 updates the same parameters. Core reading uses the static trace below. Running the code requires Primer-PY and a local CPU environment, not a GPU, model account or paid API.

20.1 Freeze the task before creating tensors

The project uses mini-gpt-corpus-v1: 24 original training records and 8 validation records, already split by whole record. They contain the France-capital, red/blue key, fictional 600/750-yuan policy and fictional order-query patterns. Some templates are very similar; the split is a small development exercise, not an independent language benchmark. All raw records are visible in data/mini-gpt/corpus.json. Nothing queries a live order system.

Use Chapter 19’s book-byte-bpe-v1, with vocabulary size 291, no normalization, BOS=256, EOS=257 and PAD=258. The tokenizer was fitted only to training records. One record occupies one context window. We never concatenate records or let a window cross the split. There are 64 slots; a record longer than the supported window raises an error instead of being silently truncated. These fixed records require at most 34 scored positions, including EOS; padding fills the unused slots.

The model has one pre-norm block, width 16, two heads of width 8, a ReLU FFN of width 64, learned absolute positions, and a final LayerNorm before an untied vocabulary head. Epsilon is 0.00001 and dropout is zero. Parameters are float64 CPU tensors. Initial matrix entries are drawn from a zero-mean normal distribution with standard deviation 0.02 using seed 1729; biases start at zero, normalization scales at one. These choices define a small teaching model, not the architecture of a commercial GPT product. The complete frozen configuration is data/mini-gpt/config.json.

20.2 Shift the target once, then pad the unused slots

Example 20.1 — Two records become a batch. Take these two training records, retaining every space and punctuation mark:

The key is red. Answer: red.
The key is blue. Answer: blue.

For the red record, byte BPE gives 14 ordinary tokens. Put BOS before them and EOS after them. The input x is that list without its last element; target y is the same list without its first element. Table 20.1 displays the whole resulting record. Indices are zero-based to match the code. The ␠ symbol in the piece column displays an ordinary space; it is not part of the raw text or an extra token.

Table 20.1 — Every real input and its next-token target.

Position index	Input ID	Input piece	Target ID	Target piece
0	256	BOS	84	T
1	84	T	265	he␠
2	265	he␠	268	key
3	268	key	32	␠
4	32	␠	277	is␠
5	277	is␠	270	red
6	270	red	261	.␠
7	261	.␠	65	A
8	65	A	276	ns
9	276	ns	119	w
10	119	w	262	er
11	262	er	259	:␠
12	259	:␠	270	red
13	270	red	46	.
14	46	.	257	EOS

There are 15 scored targets, not 14: ending the record is also predicted. BOS is known input and never a target; EOS is a target here and is not included as another scored input. Slots 15–63 contain PAD in x and the ignore sentinel −100 in y. That sentinel is a loss instruction, not a vocabulary ID. A separate Boolean array valid marks the 15 real query/key positions.

The blue record has 17 scored positions because this small tokenizer splits each blue into blu and e. Its complete real input IDs are [256,84,265,268,32,277,274,101,261,65,276,119,262,259,274,101,46]; its targets are [84,265,268,32,277,274,101,261,65,276,119,262,259,274,101,46,257]. Thus the batch has x, valid and y shapes [2,64], but only 15+17=32 scored events. Averaging over 128 padded slots would change the objective.

In batch_records, this operation is the project’s dataset interface: turn an already selected record list into aligned input, visibility and target arrays. Selecting a training mini-batch chooses complete records, not arbitrary pieces from validation. No large framework dataset object is needed to explain that contract.

20.3 Trace the structure from IDs to vocabulary scores

1 · batch_records

1. Text pieces2 complete records
2. An operationFrozen byte BPE + one target shift
3. Row numberids / valid / targets [2, 64]

15 + 17 valid targets; padding contributes no loss.

2 · MiniGPT.forward

1. Stored model numbersToken table [291, 16]
2. Stored model numbersPosition table [64, 16]
3. A list of numbersLookup + position → X [2, 64, 16]

Integer IDs choose rows; addition happens between coordinate vectors.

3 · Block.forward: communication

1. An operationRowNorm → Q/K/V [2, 2, 64, 8]
2. An operationQKᵀ / √8 + visible keys → A [2, 2, 64, 64]
3. A list of numbersAV → join → output [2, 64, 16]

Only earlier/current valid keys contribute. Softmax normalizes the last source-position axis.

4 · Block.forward: preserve and transform

1. A list of numbersR = X + attention [2, 64, 16]
2. An operationRowNorm → FFN 16 → 64 → 16
3. A list of numbersY = R + FFN [2, 64, 16]

Both residual additions preserve shape. ReLU lies between the two position-wise FFN matrices.

5 · MiniGPT.forward: score candidates

1. An operationFinal RowNorm [2, 64, 16]
2. Stored model numbersLM matrix [16, 291] + bias [291]
3. A list of numbersRaw logits [2, 64, 291]

Every column has the frozen vocabulary ID meaning; this axis is not the attention key axis.

6 · Cross-entropy

1. Row number32 real next-token IDs
2. An operationSum 32 negative log probabilities / 32
3. A list of numbersInitial mean loss = 5.670999

Ignore targets marked −100. The diagram reaches a score; the training loop must still update parameters.

Figure 20.1 — From two text records to 32 scored events Original teaching diagram, CC BY-SA 4.0. Values and rules are declared in the chapter and shared data.

Figure 20.1 maps each operation to its code name. Lookup turns integer IDs into floating-point coordinates. Position addition supplies slot information. The block communicates only through allowed attention paths and returns the same width. The final head maps that width to vocabulary candidates. There are two different probability axes: attention normalizes source positions, whereas the output distribution normalizes vocabulary IDs.

Table 20.2 — The actual MG-1 shape log. B=2, T=64, C=16, H=2, d=8, F=64, V=291.

Trace entry / operation	Shape
Input IDs, valid flags, shifted targets	[2,64] each
token_embedding lookup	[2,64,16]
position_embedding rows	[64,16], shared across examples
block_input, norm1	[2,64,16] each
q_projection, k_projection, v_projection	[2,64,16] each
q_heads, k_heads, v_heads	[2,2,64,8] each
scores	[2,2,64,64]
visible	[2,1,64,64], shared across heads
weights	[2,2,64,64]
mixtures	[2,2,64,8]
joined, attention_output, residual1, norm2	[2,64,16] each
ffn_linear, ffn_relu	[2,64,64] each
ffn_output, block_output, final_norm	[2,64,16] each
logits	[2,64,291]
Mean loss over 32 valid targets	Scalar

Q/K/V and output projections use matrices [16,16]. Splitting two heads reshapes coordinates and moves the head axis, as in Chapter 16. The scale divisor is the square root of 8, not 16 or 64. The singleton head dimension in visible means both heads receive the same causal/key-padding relation. Invalid queries are explicitly assigned zero attention mixtures; unused later logits do not contribute to loss.

A block returns [B,T,C], so the output layer must still map C to V. Here it computes:

$$Z=GW_{\text{LM}}+b_{\text{LM}},\qquad [2,64,16]\ @\ [16,291]\ +\ [291]\ \longrightarrow\ [2,64,291].$$

G denotes the final normalized representation, with shape [B,T,C]. W_LM’s columns are candidate IDs in the frozen order. The output bias has one entry per candidate. The embedding table [291,16] and W_LM are separate parameters in this project, although some language models share their corresponding weights.

Table 20.3 — Count every stored scalar parameter. These counts exclude temporary activations and optimizer state.

Component	Count
Token table, 291 × 16	4656
Position table, 64 × 16	1024
Q/K/V plus output matrices, 4 × 16 × 16; no biases	1024
FFN, 16 × 64 + 64 + 64 × 16 + 16	2128
Three LayerNorms, each 16 scales + 16 offsets	96
LM matrix and bias, 16 × 291 + 291	4947
Total	13875

This is a count derived from shapes and checked against registered parameters. It is not a full memory estimate: training also needs gradients, optimizer accumulators and intermediate arrays.

20.4 Follow real numbers through one prediction

Example 20.2 — Position 5 predicts red before seeing it. Use the initial seeded model, before any update. In the first record, input index 5 is ID 277, is␠; target ID 270 is red. Allowed input indices are 0–5. The red token at index 6 and the later Answer: red. are both forbidden at this prediction.

The first token-embedding coordinate is 0.01821338003888261; the first position coordinate is 0.027941533876330877. Their sum is 0.04615491391521349. All sixteen coordinates of that input row participate in LayerNorm: mean 0.005044787486780672, variance 0.0007407640671898172. Subtract that mean from the first coordinate and divide by the square root of variance plus 0.00001 to obtain approximately 1.500366. Normalizing just the displayed first few coordinates would be a different computation.

To check the mean and variance with all coordinates, here is that complete input row, rounded to nine decimals. Line breaks only make the one 16-coordinate row easier to read; they do not create four positions. Rounding can change the final digits of a recomputation.

[
  0.046154914, -0.015372816, -0.013930337, 0.012018233,
  0.030313746, -0.005942124, 0.044805955, 0.028551791,
  -0.052296434, 0.027663488, -0.034449716, -0.024532425,
  0.018037712, 0.011286233, 0.004664645, 0.003743734
]

Table 20.4 — Every representation stage at this position. The first three coordinates are displayed to six decimals; full selected rows are in data/mini-gpt/forward.json. The FFN middle rows have 64 coordinates; all other representation rows below have 16.

Stage	First three coordinates
Token embedding	[0.018213,0.011841,-0.009374]
Position embedding	[0.027942,-0.027214,-0.004556]
Sum / block input	[0.046155,-0.015373,-0.013930]
First norm	[1.500366,-0.745166,-0.692521]
Q projection	[0.086787,-0.040868,-0.193362]
K projection	[-0.013506,0.011333,-0.061313]
V projection	[0.007704,-0.032708,0.052636]
Joined head mixtures	[0.035102,0.011446,-0.001279]
Attention output projection	[0.002184,0.001408,-0.002312]
First residual	[0.048339,-0.013964,-0.016242]
Second norm	[1.552943,-0.721283,-0.804429]
First FFN transform	[0.183987,0.108629,-0.107695]
ReLU	[0.183987,0.108629,0.000000]
Second FFN transform	[-0.005631,-0.000809,0.002055]
Second residual / block output	[0.042707,-0.014773,-0.014188]
Final norm	[1.401739,-0.594476,-0.574142]

The table is a measured forward trace, not hand-assigned activations. For example, the first residual adds 0.04615491391521349 and 0.0021837806813666213. The second adds −0.005631200185285851 to that result, giving 0.042707494411294254. ReLU’s removal of the negative third FFN coordinate is visible in the middle; a successful shape log alone would not establish that it occurred.

The first head’s six allowed scaled scores and weights account for the between-position calculation omitted from the coordinate table:

Table 20.5 — One query reads its six known input positions. All indices 6–63 have weight exactly zero for this query.

Key index	Scaled score	Weight
0	-0.005191	0.166037
1	-0.006729	0.165782
2	-0.003865	0.166257
3	0.001567	0.167163
4	0.001338	0.167125
5	0.004393	0.167636

The scores are initially similar, so the mixture is nearly uniform over those six positions. The scalar-loop comparison takes the actual eight-coordinate queries/keys, recomputes their dot products and value sums, and agrees with the tensor implementation within 0.000000000001. There is no claim that this untrained head has found a meaningful key clue.

20.5 Compute loss over candidates, then over valid events

At this position, the model returns 291 raw logits. Candidate 270 has logit 0.09915046290174213. The largest logit in the row is 0.20936204844775802; subtracting it from all candidates gives an exponentials sum of 236.07843517556194. Chapter 6’s rule therefore gives:

$$p_{270}=\frac{\exp(0.09915046290174213-0.20936204844775802)}{236.07843517556194} \approx0.003793843386,\qquad \ell=-\ln p_{270}\approx5.574375688.$$

The 291 candidates, including all ordinary bytes/pieces and special entries, constitute the normalization denominator. BOS and PAD are never correct targets in this dataset, but their output columns still exist during loss calculation. Generation will separately forbid selecting them. Attention’s six visible keys are not the six candidates in this denominator.

For the whole two-record batch, sum the 32 valid target losses and divide by 32. The observed mean is 5.670998865509144 nats per target. An independent scalar calculation from the returned logits gives 5.670998865509145. This batch mean need not equal the loss of the one red prediction, or the full training-set mean used in the next chapter.

A second check has a known answer before running the code. Set every vocabulary logit to zero. All 291 exponentials are 1, so each candidate gets probability 1/291; every valid target has loss $\ln(291)\approx5.673323267$. Changing the number of ignored padding slots must not change that mean. The implementation passes this analytic check too. Merely matching a prerecorded model loss would be a weaker check because that number might itself have come from a mistaken implementation.

20.6 Let code verify the structure

code/mini-gpt/model.py implements the calculation with PyTorch tensors and automatic differentiation. It does not call nn.Transformer, a pretrained model loader, or a packaged attention block. Matrix products, coordinate normalization, masks, head reshaping, ReLU and residual additions are explicit. The model is small enough to inspect alongside Figure 20.1.

One additional Python idea is useful. A class describes how to create an object containing data and functions; MiniGPT(cfg) creates one such object. self.token_embedding is a tensor owned by that instance. Inheriting from nn.Module lets PyTorch collect tensors marked nn.Parameter and the parameters of its submodules. Calling model(ids) invokes its forward method. That call computes with existing parameters; it does not train them or fetch a model from elsewhere. The named RowNorm, Block, and MiniGPT classes organize the already explained arithmetic.

After environment setup in code/part-iv/README.md, use that environment’s Python interpreter to run the complete MG-1 verification from the book root:

python code/mini-gpt/forward.py

The program selects the two stated records and follows this call chain:

cfg = config()
tokenizer = load(ROOT)
model = create_model(cfg)
ids, valid, targets = batch_records(records, tokenizer, cfg["context_length"])
model.eval()
with torch.no_grad():
    logits, loss, trace = model(ids, valid, targets, trace=True)

IDs and targets are torch.long integers; parameters and representations are float64 CPU arrays. eval() selects evaluation behavior, while no_grad() avoids building a gradient graph. With zero dropout and row LayerNorm, this model’s arithmetic does not change between train/eval, but the separate meanings matter when extending it. The loss function receives raw logits; no extra Softmax is applied before cross-entropy.

Behavior checks in test_model.py alter future tokens, padding contents and batch companions and require real prefix logits to stay unchanged where the contract demands it. They test target shift and EOS, scalar attention agreement, analytic uniform loss, exact parameter count, and rejected empty or overlong input. Backward/update checks then establish that gradients reach the embeddings and block parameters. These are local implementation checks, not evidence of language competence.

20.7 MG-1 is a functioning computation, not a trained ability

MG-1 now has a fixed raw-text dataset, tokenizer identity, complete structural diagram, shape log, numerical trace, logits, loss and CPU checks. It is the neural continuation of the earlier hand-count language-model project. Compared with Chapter 4’s count table, this model shares parameter tables and transformation matrices across positions; its loss can train those parameters.

The untrained loss near the uniform baseline is unsurprising. Computing a loss tells us how the current parameters score the targets; it does not change them. Chapter 21 will run backward, update these same 13,875 parameters, save their training state, and ask what the changed outputs actually demonstrate.

20.8 Exercises

1. For the red record in Table 20.1, how many input slots are real, how many are padding, and how many targets enter the mean? What error results from shifting y twice?
2. Explain the two different 64 axes in attention scores [2,2,64,64], and why logits [2,64,291] have a different final axis.
3. If every logit were zero and only two valid targets remained, compute the expected mean loss. What would dividing their summed loss by 128 report instead?
4. Remove the output projection after concatenating two heads of width 8. Would the residual shape still match width 16? Why does this not prove the model is unchanged?
5. Add one new vocabulary entry while keeping width 16 and the existing untied matrices and bias policy. How many parameters must be added? What else must be versioned?
6. A program passes all shapes but its earlier logits change when a future token is replaced. Give two possible paths for the leak and state what MG-1 evidence must be rechecked.

20.9 Reference answers

1. There are 15 real slots, 49 padding slots and 15 scored targets. EOS contributes the fifteenth event. A second shift would align each input with a target two steps ahead instead of the declared next token; compatible shapes would not fix the wrong task.
2. The first attention 64 is the receiving query position; the second is the contributing key position. Scores are separate for two examples and two heads. Logits instead give 291 candidate IDs at each of 64 output positions. One Softmax mixes sources; the other distributes probability over possible next tokens.
3. The correct mean remains $\ln(291)\approx5.673323267$. Dividing two losses by 128 would report $\ln(291)/64\approx0.088645676$, misleadingly small because 126 unscored placeholders were counted in the denominator.
4. Yes: two widths of 8 already concatenate to 16. But the learned W_O combines those coordinates before the residual. Removing it replaces that transformation with an identity map and generally changes the function despite matching shapes.
5. Add 16 embedding entries, 16 output-matrix entries and one output bias: 33 parameters. Version the ordered vocabulary, merges/special-token policy, tokenizer identity, data encoding and model/checkpoint configuration. Existing rows must retain their meanings or be deliberately migrated.
6. Attention may use an absent/reversed causal mask, or normalization/preprocessing may mix future positions before attention. Recheck visible relations, actual prefix perturbations, coordinate-only norms and target construction. A complete diagram and numerical trace should identify the permitted path; shapes alone cannot establish it.

20.10 References

• Vaswani et al. (2017), Attention Is All You Need, Section 3; Xiong et al. (2020), On Layer Normalization in the Transformer Architecture, Table 1. Locate the operations combined here and the chosen pre-norm order; the small parameter choices and data are original.
• PyTorch, Module, 2.7 and cross_entropy, 2.7. Explain parameter registration, module calls, raw-logit input and ignored-target reduction used in the executable checks.