Chapter 5 Batches, Shapes, Vectors, and Matrices

5.1 How can one calculation serve many text positions?

Chapter 3 looked up one row of numbers for each token ID. In its teaching table, ID 4, ␠France, retrieves [2,1]. The space marker belongs to the token’s spelling; it is not another coordinate. Neither coordinate has a natural meaning such as nationality or importance. They are the two numbers assigned to that row.

One row was enough to explain lookup. A sentence needs several rows, and a training step usually uses several sentences. How can we keep their identities while applying the same calculation to all of them? Our criterion is stronger than “the program runs”: we should be able to locate any result, name the input entries that produced it, and explain what its axes mean.

First arrange positions vertically. Take two short fragments from the existing table. These are fragments for arithmetic, not two complete claims about France.

Table 5.1 — From IDs to two small matrices. All vector values are author-assigned, dimensionless teaching values from Chapter 3. Indexing starts at zero.

Sample	Position	Token ID and spelling	Coordinate 0	Coordinate 1
0	0	4, ␠France	2	1
0	1	5, ␠is	1	−1
1	0	6, ␠Paris	2	2
1	1	7, .	0	−1

For sample 0, the rows form the rectangular table $X_0=\begin{bmatrix}2&1\\1&-1\end{bmatrix}$. A rectangular numerical table is a matrix. It has two rows and two columns, so its shape is [2,2]. Here the first axis counts positions and the second counts coordinates. Shape records the sizes in their declared order; it does not record what the rows and columns mean.

Now place the two matrices in an ordered collection called a batch. Write the whole input as $X=[X_0,X_1]$, where $X_1=\begin{bmatrix}2&2\\0&-1\end{bmatrix}$. Its shape is [2,2,2]: two samples, two positions per sample, two coordinates per position. The more informative name is [B,T,C], with $B$ the batch size, $T$ the number of positions in each stored sample, and $C$ the coordinate count. These letters are conventions, not operations.

To read $X_{1,0,1}=2$, take sample 1, its position 0 (␠Paris), then coordinate 1. There are $B\times T\times C=8$ stored numbers. The ID array, by contrast, has shape [B,T]=[2,2] and contains four labels. Adding the coordinate axis is the work of lookup, not a property of an ID’s numerical magnitude. A numerical array with any number of axes is often called a tensor in machine-learning software. For this chapter, that word adds no new arithmetic: a vector has one axis, a matrix two, and this batch three.

5.2 Calculate one output cell before the whole table

Suppose each row should produce three candidate scores in the fixed order Paris, Lyon, other. “Other” groups all alternatives for this teaching calculation; this is not the full vocabulary or a trained predictor. We supply two coefficients for each score because the input row has two coordinates.

Example 5.1 — One row becomes three scores. The input is $x=[2,1]$. The coefficient columns are $[1,0]$, $[0,1]$, and $[-1,1]$. Multiply matching coordinates, then add: the first score is $2\times1+1\times0=2$; the second is $2\times0+1\times1=1$; the third is $2\times(-1)+1\times1=-1$. A negative score is allowed. We have not produced probabilities.

Each multiply-and-add is the dot product introduced in Chapter 3. For two length-$C$ lists $x$ and $v$, it is

$$x\cdot v=\sum_{c=0}^{C-1}x_c v_c.$$

The summation sign means “add one product for each coordinate.” The subscript $c$ selects the same coordinate from both lists; the final index is $C-1$ because numbering begins at zero. With $C=2$ and $v=[-1,1]$, the expression expands to the two products $2(-1)+1(1)=-1$. Equal lengths permit this operation, but corresponding entries also need compatible meanings and units.

Put the three coefficient columns side by side:

$$W=\begin{bmatrix}1&0&-1\\0&1&1\end{bmatrix}.$$

$W$ has shape [C,K]=[2,3], where $K$ is the number of output score channels. Its rows correspond to input coordinates; its columns correspond to candidates. Applying those same column rules to every row is matrix multiplication, often abbreviated MatMul. For sample 0 the complete result is

$$X_0W= \begin{bmatrix}2&1\\1&-1\end{bmatrix} \begin{bmatrix}1&0&-1\\0&1&1\end{bmatrix} = \begin{bmatrix}2&1&-1\\1&-1&-2\end{bmatrix}.$$

Check the bottom-right cell: it uses the second input row and third coefficient column, so $1(-1)+(-1)(1)=-2$. It does not multiply entire rows element by element and leave two answers. The coordinate axis is multiplied and summed away; each input row now has three scores.

Table 5.2 — Complete batched output, with the candidate order attached.

Sample	Position	Paris score	Lyon score	Other score
0	0	2	1	−1
0	1	1	−1	−2
1	0	2	2	0
1	1	0	−1	−1

The general rule for this particular batch operation is

$$Z_{b,t,k}=\sum_{c=0}^{C-1}X_{b,t,c}W_{c,k},\qquad [B,T,C]\ @\ [C,K]\ \longrightarrow\ [B,T,K].$$

Here $b$ chooses the sample, $t$ the position, and $k$ the output candidate. Only $c$ is summed. Thus $Z_{1,0,2}=2(-1)+2(1)=0$, the third score for ␠Paris. The @ symbol is the matrix-multiplication operator in the accompanying Python practice. The same $W$ is reused for every sample and position; batching neither creates a separate parameter matrix per sentence nor lets one sample read another. NumPy’s matmul treats the final two axes as matrix axes and supports stacks of matrices; this example uses a shared two-dimensional right operand. NumPy: matmul

Input X: [B,T,C] = [2,2,2]

1. A list of numbersSample 0: [2,1]; [1,−1]
2. A list of numbersSample 1: [2,2]; [0,−1]

Choose a sample, then a position, then a coordinate.

Shared W: [C,K] = [2,3]

1. Stored model numbersRows: [1,0,−1]; [0,1,1]
2. An operationFirst row: 2×[1,0,−1] + 1×[0,1,1]

Sum over C only. Do not combine samples or positions.

Output Z: [B,T,K] = [2,2,3]

1. A list of numbersSample 0: [2,1,−1]; [1,−1,−2]
2. A list of numbersSample 1: [2,2,0]; [0,−1,−1]

K names the ordered candidates Paris, Lyon, other. These scores are not probabilities.

Figure 5.1 — One shared matrix changes coordinates into score channels; samples and positions stay separate. Original diagram and teaching values, CC BY-SA 4.0.

Figure 5.1 separates the axis that disappears from the two axes that survive. Follow the first input row through the middle calculation to its output. The double border marks supplied parameter values. It does not claim these particular values were learned.

5.3 Turning a table is different from relabeling its contents

We wrote coefficient vectors as columns so that an input row could meet each one. Another file might store one candidate per row. We need to exchange the row and column roles while preserving which number belongs to each pair. That operation is a transpose:

$$W^{\mathsf T}=\begin{bmatrix}1&0\\0&1\\-1&1\end{bmatrix}, \qquad (W^{\mathsf T})_{k,c}=W_{c,k}.$$

The superscript $\mathsf T$ here means transpose; it is not the position-count letter $T$ in [B,T,C]. The original shape [2,3] becomes [3,2]. For example, the −1 at row 0, column 2 moves to row 2, column 0. Transposing twice recovers the original table.

This also lets us compare all input rows with a collection of reference rows. If $R$ stores $K$ reference vectors as rows with shape [K,C], then $X_0R^{\mathsf T}$ has shape [T,K]. Cell $(t,k)$ is the dot product of input row $t$ and reference row $k$. Transposition aligns coordinate axes; it does not establish that the references use the right vocabulary or measurement system.

For an array with three axes, the exchange must name the axes. Changing [B,T,C] into [T,B,C] moves sample and position roles. In our example both sizes happen to be 2, so the printed shape remains [2,2,2]. Nevertheless, the item at [0,1] changes from [1,−1] to [2,2]. A shape assertion alone cannot detect that mistake. A reshape, which regroups a stored sequence of numbers into new axis lengths, is not in general a substitute for an axis permutation. The NumPy exercise checks actual entries as well as sizes; the official transpose documentation specifies the axis mapping. NumPy: transpose

5.4 Similarity needs a denominator and a boundary

A large dot product can arise because directions align, because vectors are long, or both. Chapter 3 divided by both lengths to isolate their angle. We can now follow that denominator with the same rows used above.

Example 5.2 — Compare two directions. Let $u=[2,1]$ for ␠France and $v=[2,2]$ for ␠Paris. Their dot product is $2(2)+1(2)=6$. The length of $u$ is $\sqrt{2^2+1^2}=\sqrt5$; the length of $v$ is $\sqrt{2^2+2^2}=\sqrt8$. The square root recovers a length after adding squared coordinate contributions. Dividing gives

$$\cos(u,v)=\frac{u\cdot v}{\|u\|\|v\|} =\frac6{\sqrt5\sqrt8} =\frac6{\sqrt{40}}\approx0.948683.$$

The vertical bars denote vector length, not a word count. If we instead use $r=[1,-1]$, then $u\cdot r=1$, $\|r\|=\sqrt2$, and the cosine is $1/\sqrt{10}\approx0.316228$. Under these supplied coordinates, the first pair has closer directions. Nothing in the arithmetic proves the words are equivalent; even France and Paris name different kinds of things.

Multiply $v$ by 10. The dot product becomes 60 and its length becomes $10\sqrt8$, so the cosine stays the same. This is useful when overall magnitude is irrelevant to the comparison, and harmful when magnitude carries information we need. If a vector is [0,0], its length is zero and the cosine denominator is zero. The angle is undefined. Our general arithmetic function returns None for that case; a retrieval application must separately decide whether to abstain. It must not silently call an undefined angle evidence of semantic agreement.

5.5 Rectangular storage does not make padding into text

Real samples have different lengths. Shorten sample 1 to just ␠Paris, keeping sample 0 unchanged. A rectangular array still needs a second row for sample 1. One storage convention inserts a zero row and separately records the real lengths [2,1]. This inserted row is padding.

Example 5.3 — An innocent-looking extra row changes an average. Sample 1 now stores [[2,2],[0,0]]. Its real text has one row, so the mean representation is $[2/1,2/1]=[2,2]$. Summing both stored rows and dividing by the storage length 2 instead gives [1,1]. The padded sum happens to be harmless here; the wrong denominator halves the result. If two samples are supposed to receive equal weight, this can change their relative contribution to a later calculation.

The padding row is bookkeeping, not the unknown-token row from Chapter 3, even though both happen to contain zeros. An unknown token occupies a real text position; padding does not. A zero input also need not stay zero: an operation $xW+b$ with nonzero added vector $b$ produces $b$ at a padded row. Therefore, “I used zeros” is not a complete policy for excluding padding from pooling or loss. Store lengths or a validity indicator and use it where needed. The complete attention-mask mechanism belongs to Chapter 16; the necessary idea here is simply that stored slots and genuine observations are different counts.

Table 5.3 — Three errors that compatible shapes cannot reject.

Unchanged shape	Changed meaning	Observable consequence
Output [2,2,3]	Consumer reads columns as Lyon, Paris, other	The score 2 in the first row is attributed to the wrong candidate
Input [2,2,2]	Sample and position axes are exchanged	A later operation combines different text positions or samples
Two length-2 vectors	One source measures amounts in yuan, another in hundreds of yuan	A dot product mixes scales unless coordinates and coefficients are converted consistently

These failures have different remedies: version the candidate mapping, name and test axes, or attach and convert units. Merely adding another shape assertion fixes none of them. Arrays provide a compact way to organize arithmetic; the data contract supplies its meaning.

5.6 Practice after the Python bridge

The core chapter ends without requiring code. The practice order is now: Python and Notebook Foundations → the parameterized Chapter 4 N-gram notebook → the NumPy shape exercise → Chapter 7’s training practice. The primer teaches the syntax needed to follow those files. Chapter 4’s paper argument and MG-0 project milestone remain usable without Python.

The companion files are notebooks/part-ii/01-primer-and-ngram.ipynb, 02-shapes-and-loss.ipynb, and code/part-ii/shape_practice.py. They read the existing Part I fixtures and data/part-ii/arithmetic.json. The shape exercise produces [2,2,2] @ [2,3] → [2,2,3] and exactly Table 5.2. It also exchanges the first two axes and demonstrates equal sizes with unequal contents. These are checks of specified arithmetic, not measurements of model quality. Exact run conditions and saved results belong to the companion practice records.

5.7 Chapter summary

A batch adds a sample axis to the position and coordinate axes. Matrix multiplication forms each output by a row-column dot product and sums over the shared coordinate axis. Transposition moves entries with their axis roles; it is not merely changing a displayed shape. Cosine removes positive scale through its denominator and is undefined for a zero vector. Padding makes storage rectangular without creating an observation.

We can now produce one score vector for every position in every sample. The scores can be negative, and their sum need not be one. Chapter 6 asks how to turn them into probabilities and into a measurable prediction error.

5.8 Exercises

1. Transfer the multiplication. Use $A=\begin{bmatrix}1&2\\0&-1\end{bmatrix}$ and the same $W$ as Example 5.1. Compute all six output cells. If matrix $A$ is one sample, what are $B,T,C,K$ and the batched output shape?
2. Repair the labels. An output row is [2,1,−1] in Paris, Lyon, other order. A consumer expects Lyon, Paris, other. Can you repair the interface by transposing the batch? Supply the correctly reordered row and explain why the original shape check passed.
3. Choose a denominator. Two real rows are [2,0] and [0,2]; a third stored row is padding [0,0]. Calculate the valid-position mean and the storage-position mean. State what extra information distinguishes the two.
4. Test a similarity claim. Compare [2,1] with [20,10] and with [0,0]. A colleague claims the first result proves the represented texts are interchangeable. Which conclusions does the calculation support?
5. Audit an axis exchange. For Table 5.1, write the two rows at first-axis index 0 after exchanging $B$ and $T$. Explain why a test that checks only [2,2,2] misses the change.

5.9 Reference answers

1. Row 0 gives $1(1)+2(0)=1$, $1(0)+2(1)=2$, and $1(-1)+2(1)=1$. Row 1 gives 0, −1, −1. Thus $AW=\begin{bmatrix}1&2&1\\0&-1&-1\end{bmatrix}$. Adding a sample axis gives $B=1,T=2,C=2,K=3$ and output [1,2,3]. The 2 that disappears counts input coordinates, not samples.
2. Reorder the candidate axis to [1,2,−1] and update the mapping contract. Transposition exchanges axes, whereas this repair permutes entries within the candidate axis. Both vocabularies have three entries, so the old and new shape are identical.
3. The sum is [2,2]. Dividing by two genuine positions gives [1,1]; dividing by three stored positions gives [2/3,2/3]. A real length of 2 or a validity list [true,true,false] provides the missing distinction. The zero row alone cannot say whether it is real data.
4. The first pair has dot product 50, lengths $\sqrt5$ and $10\sqrt5$, and cosine 1. They have the same direction with different magnitudes under the chosen representation. Interchangeability is a separate task claim. The zero vector gives a zero denominator, so its cosine is undefined rather than a second similarity verdict.
5. The first new slice contains old $X_{0,0}=[2,1]$ and old $X_{1,0}=[2,2]$: position 0 from each sample. The old first slice contained both positions of sample 0. The sizes match because $B=T=2$; an entry-level assertion or an axis contract is needed to catch the mix-up.

5.10 References

• NumPy, matmul. Specifies matrix and stacked-array multiplication, dimension matching and the @ operation used in the exercise. Our complete numerical example is original.
• NumPy, transpose. Specifies axis permutations and explains why transposing a one-dimensional array does not by itself create a column matrix. The practice pins its own dependency version separately from this maintained documentation.