Chapter 7 The Smallest Machine-Learning and Neural-Network Training Loop

7.1 What would make the next prediction different?

A model can compute logits, probabilities and loss without learning anything. If the parameters stay unchanged, repeating the same input gives the same calculation. Chapter 6 supplied the error signal; now we need an operation that uses it to change stored numbers.

Start with a small classification task. An input is a point with two dimensionless coordinates, $x=[x_0,x_1]$. The target is either class 0 or class 1. This is an authored numerical task for studying training, not a classifier of people or a live travel-policy system. We first use one point, then ask what happens on points excluded from parameter fitting. Success on that second question is the reason to distinguish learning a useful pattern from merely fitting observed labels.

Example 7.1 — Inspect a small parameter change before naming a gradient. Take $x=[1,2]$ with target $y=1$. Supply weights $w=[0,0]$ and bias $b=0$. A bias is an added adjustable constant: unlike a weight, it does not multiply an input coordinate. Define a class-1 score $z=x_0w_0+x_1w_1+b$, and use class logits [0,z] in class 0, class 1 order. Softmax gives

$$p=P(y=1\mid x)=\frac{e^z}{1+e^z}=\frac1{1+e^{-z}}.$$

This two-class form is called the sigmoid. At the supplied parameters, $z=0$, $p=1/2$ and the target loss is $\ell=-\ln p=\ln2\approx0.693147$. The target value 1 identifies a class; it does not mean the input coordinate is 1.

Keep everything fixed except $w_0$. Raising it from 0 to 0.01 raises $z$ to 0.01 and lowers the loss to approximately 0.688160. Lowering $w_0$ to −0.01 raises loss to approximately 0.698160. Near this starting point, increasing $w_0$ helps this example. We discovered a direction by comparing actual values, without yet having a rule for millions of parameters.

7.2 A local rate lets the effect travel through the calculation

The loss change divided by the parameter change estimates a local rate of change. Using the two perturbations above gives

$$\frac{0.6881596805-0.6981596805}{0.01-(-0.01)}\approx-0.5.$$

The negative sign says that a small increase of this weight lowers loss. The magnitude means approximately half a nat less loss per unit increase of $w_0$ near the current point. It does not predict the effect of an arbitrarily large change.

For a smooth operation, as the change is made smaller this ratio approaches its derivative. We write $\partial\ell/\partial w_0$ when varying $w_0$ while holding the other independent inputs and parameters fixed. The symbol names the local sensitivity; it is not another parameter in the model. We can compute it efficiently by passing local sensitivities through the operations already used in the forward calculation.

We can inspect intermediate rates in the same way. At probabilities 0.499 and 0.501, negative log loss is approximately 0.695149183231 and 0.691149177897. The loss difference divided by $0.501-0.499=0.002$ is approximately −2.000003. At scores −0.01 and 0.01, sigmoid gives probabilities approximately 0.497500020833 and 0.502499979167; their difference divided by 0.02 is approximately 0.249998. Smaller intervals approach −2 and 1/4 at the central point. These numbers describe sensitivity to the intermediate probability and score, respectively, before we connect either one to a weight.

For this target-1 example, three relationships suffice. Near $p=1/2$, the slope of $-\ln p$ is $-1/p=-2$. The sigmoid’s slope is $p(1-p)$, here $1/4$: a small change of 0.01 in $z$ changes $p$ by about 0.0025. Finally, the score is a weighted sum, so increasing $w_0$ by a small amount increases $z$ by that amount times $x_0=1$.

Why multiply these rates? A small weight change $\Delta w_0$ gives approximately $\Delta z=1\Delta w_0$; this gives $\Delta p\approx(1/4)\Delta z$; then $\Delta\ell\approx(-2)\Delta p$. Combining them gives $\Delta\ell\approx(-1/2)\Delta w_0$. This is the chain rule: follow a route through intermediate values and multiply its local rates. If a parameter influences loss through several routes, add those routes’ contributions.

Table 7.1 — All local values for one backward pass.

Relationship	Local rate at the starting point	Consequence
$\ell=-\ln p$	$\partial\ell/\partial p=-2$	Increasing target probability lowers loss
$p=1/(1+e^{-z})$	$\partial p/\partial z=1/4$	A small score change is reduced here
$z=x_0w_0+x_1w_1+b$	Rates for $w_0,w_1,b$ are $1,2,1$	The second coordinate contributes twice the sensitivity
Chain from score to loss	$(-2)(1/4)=-1/2$	$\partial\ell/\partial z=-1/2$
Chain to all parameters	$[-1/2,-1,-1/2]$	Rates for $[w_0,w_1,b]$ in that order

The derivatives of log and sigmoid can be derived from their functions; the local numerical checks here make their role visible without requiring a prior calculus course. For a binary target $y$ equal to 0 or 1, the combined score derivative is $p-y$ when the loss scores that target. Substituting $p=1/2,y=1$ recovers −1/2; for $y=0$ it becomes +1/2 and asks us to reduce the class-1 score instead.

The list of derivatives with respect to the parameters is the gradient. Computing it from loss back through stored intermediate values is backpropagation, or the backward pass. An automatic-differentiation system records the operations and applies their derivative rules. It does not discover the correct labels or decide what loss should represent. Nor does computing a gradient by itself change a weight. PyTorch: automatic differentiation

7.3 Close the loop with an actual update

The local rate tells us which direction raises loss. To seek a reduction, move in the opposite direction. Gradient descent uses a positive step size $\eta$, called the learning rate:

$$\theta_{\rm new}=\theta_{\rm old}-\eta\,\nabla_\theta\ell.$$

$\theta$ collects the parameters; $\nabla_\theta\ell$ collects the corresponding local rates in the same order. The subtraction is entry by entry. This is a local proposal, not a guarantee that every chosen step size will lower loss on every example.

Example 7.2 — Complete one training step. Use $\eta=0.1$ and the gradient from Table 7.1. The new values are $w_0=0-0.1(-0.5)=0.05$, $w_1=0-0.1(-1)=0.1$, and $b=0-0.1(-0.5)=0.05$. Run the same input again:

$$z_{\rm new}=1(0.05)+2(0.1)+0.05=0.3, \qquad p_{\rm new}=\frac1{1+e^{-0.3}}\approx0.574443.$$

The new loss is $-\ln0.574443\approx0.554355$, using the unrounded probability in calculation. We have followed the entire loop: forward → loss → backward → update. Forward computes intermediate values; loss compares with the target; backward computes sensitivities; update changes parameters. The next pass uses the new parameters.

Forward → loss

1. Stored model numbersw=[0,0], b=0; x=[1,2]
2. An operationz=x·w+b=0 → p=1/2
3. An operationTarget 1: L=−ln p=ln 2

Read parameters without changing them.

Backward → update

1. An operation(−2)×(1/4)=−1/2; gradients w=[−1/2,−1], b=−1/2
2. Stored model numbersSubtract 0.1×gradient: w=[0.05,0.1], b=0.05

New forward: z=0.3, p≈0.574443, L≈0.554355. Return to forward for another step.

Figure 7.1 — Forward computes values, backward computes sensitivities, update changes parameters. Original hand-derived computation graph, CC BY-SA 4.0.

Follow Figure 7.1 in both directions. The backward numbers describe how a change would propagate; they are not a second set of predictions. The double-bordered parameter boxes contain the values that persist between steps. Saving these values produces a model snapshot; that is distinct from completing the MG-0 project milestone in Chapter 4.

An optimizer implements an update rule and any state that rule needs. Plain stochastic gradient descent, or SGD, applies the subtraction above to gradients estimated from a selected training batch. A batch containing all training examples gives a full-data gradient. For mean loss, each example’s contribution is divided by the batch’s real example count before the contributions are combined. The one-point example uses one genuine observation and therefore divides by 1.

PyTorch expresses the loop as follows; the runnable setup is in code/part-ii/classifier.py and the training notebook:

optimizer.zero_grad()
logits = model(x)
loss = torch.nn.functional.cross_entropy(logits, targets)
loss.backward()
optimizer.step()

zero_grad() clears previously accumulated gradients. model(x) is forward computation. For the practice classifier, logits have shape [B,2] and integer targets shape [B]; the loss averages over $B$ genuine examples. backward() fills gradients, and step() applies the optimizer rule. Repeating backward() without the intended clearing can accumulate old contributions. Repeating forward without step() cannot train the model. During evaluation, model.eval() selects evaluation behavior and torch.no_grad() disables gradient recording; neither is an optimizer update. PyTorch 2.7: SGD

A large step can overshoot a local improvement. For the simple loss $L=(w-1)^2$ at $w=0$, the slope is −2. A step size 0.1 gives $w=0.2$ and loss 0.64, down from 1; a step size 2 gives $w=4$ and loss 9. The gradient direction was informative in both cases. The second update used that information too far from the point where it was computed.

7.4 More layers can fit more patterns—and more noise

The score $x\cdot w+b$ divides the plane by a straight line when we choose class 1 for a positive score. Suppose the label is 1 when the two coordinates have the same sign, and 0 when they have opposite signs. The four corner points [1,1], [−1,−1], [1,−1], [−1,1] then alternate classes around the origin. One straight line cannot separate the two diagonal class-1 corners from both class-0 corners.

A small neural network can first compute several weighted sums, pass them through a nonlinear function, and combine the results. Our practice uses ReLU, which maps a value $a$ to $\max(0,a)$: negative values become zero and positive values remain. An affine layer multiplies by a matrix and adds a bias. Without a nonlinear step, successive affine layers could be combined into one affine map, retaining the straight-line limitation. Nonlinearity allows more complicated boundaries; it does not ensure that the extra flexibility will be used well.

The fixed network maps [B,2] → [B,32] → [B,32] → [B,2], with ReLU after each hidden layer. “Hidden” means an intermediate representation, not an inaccessible source of knowledge. Each layer’s weights and biases are learned. The layer widths, learning rate and number of updates are hyperparameters, settings selected outside the gradient updates. The three layers have $2\times32+32=96$, $32\times32+32=1056$, and $32\times2+2=66$ trainable numbers: one weight per input/output connection and one bias per output. Their total is $96+1056+66=1218$. These 1,218 trainable numbers give this small network enough flexibility to fit a tiny set closely.

Table 7.2 — Freeze roles before fitting. The original quadrants-v1 data has 120 disjoint coordinate pairs drawn with a fixed generator, rounded to four decimal places. The clean rule is class 1 if $x_0x_1>0$.

Split	Rows	Labels and use
Training	24	Four predetermined labels flipped; all 24 rows contribute gradients
Validation	48	Clean rule labels; choose the first snapshot attaining minimum validation loss
Test	48	Clean rule labels; score the selected snapshot after selection

For example, training point [0.9927,0.7696] has label 1, and [−0.1784,−0.0846] also has label 1. Point [0.3345,−0.6350] would have rule label 0, but its training label is deliberately flipped to 1. All coordinates, labels and flip flags are saved. This construction deliberately exposes fitting of label noise; it is not a claim that training and deployment labels in general differ this way.

The split is fixed before training. Related records can leak information even when their IDs differ: near-duplicate questions or versions of the same document may need grouping or a time-based split in another task. No such grouping problem is solved merely by shuffling rows. Here the disjoint synthetic coordinates and explicit rule make the experiment small enough to audit.

7.5 Read the measured divergence between training and validation

Example 7.3 — Diagnose one actual overfit. The CPU run uses seed 17, float64, one computation thread, all 24 training rows per update, and Adam with learning rate 0.03 for 600 updates. Adam is an optimizer with moving gradient statistics; it is not the plain SGD used for the hand calculation. Its role here is a fixed, named update choice, not a new derivation. The full settings and saved outputs are shared by both language editions.

Table 7.3 — Measured snapshots from the same run. Loss is mean nats per labeled point; accuracy counts class predictions, taking the larger of the two logits.

Updates	Train loss	Train correct	Validation loss	Validation correct
0	0.697242	7/24	0.696076	19/48
5, selected	0.527363	17/24	0.581859	37/48
20	0.376093	19/24	0.736129	37/48
100	0.063092	23/24	6.900090	33/48
300	0.050499	23/24	9.593773	31/48
600	0.033060	24/24	8.150310	31/48

By the end, every training label is classified correctly, including the deliberately corrupted ones. Yet validation loss is much larger than at step 5. This widening gap is evidence of overfitting in this experiment: improving fit to the training observations has harmed prediction on the declared validation population. Validation loss need not rise at every individual step; the comparison concerns the overall divergence and the selected criterion.

Solid: training. Dashed: validation. Both axes are linear. The dotted vertical line marks the selected step 5, close to the left edge.

Figure 7.2 — Fitting the training labels eventually harms validation likelihood. Original CPU experiment and diagram, CC BY-SA 4.0; full measured points in classifier-run.json.

Figure 7.2 plots all 601 observations, including initialization, rather than drawing a schematic curve. The axes are linear. The training line falls near the bottom while the validation line climbs. The validation minimum at step 5 lies close to the left edge. Table 7.3 makes that early point readable without estimating it from pixels.

Why can validation loss increase so much while a majority of labels remain correct? Accuracy only asks which class wins. Log loss also penalizes confidently wrong predictions. For an example classified incorrectly, reducing the probability of its true target from 0.1 to 0.0001 changes its loss from about 2.303 to 9.210 without changing whether it is counted wrong. Loss and accuracy are both useful, but they summarize different behavior.

The predeclared selection rule keeps the lowest-validation-loss snapshot, step 5. Only then does the code evaluate the test split: 38/48 correct, mean loss 0.471905. There is no test curve used to choose the step. Repeatedly rerunning the same fixed code is a reproduction, not 48 new observations. Choosing a new architecture because these test results look disappointing would use the test set for development; a later honest final assessment would then require fresh held-out evidence.

This one small result does not estimate real-world language-model quality. Even the validation choice is based on only 48 points and many inspected snapshots, so it can be optimistic. The practical remedies suggested by the failure are to inspect labels, reduce unnecessary capacity, use additional appropriate training data, or select an earlier snapshot. They require new declared experiments, not a promise that one knob always cures overfitting. Generalization asks about behavior beyond the particular fitted observations. Dive into Deep Learning, §3.6

7.6 Keep three meanings of checking separate

A validation set supplies held-out examples for development choices. Schema validation checks a data contract, such as whether an order record has an order_id string and a permitted status value. A later verifier may check a particular proposed answer or action against evidence and rules. These checks do not substitute for one another. A classifier can pass a schema check while generalizing poorly; a well-performing model does not thereby gain permission to cancel an order.

The practice path remains Chapter 5 core → Python primer → Chapter 4 N-gram notebook → NumPy shape exercise → 03-training.ipynb. The notebook checks the hand-derived gradients, reproduces the fixed training run, and compares it with the saved record. classifier-data.json freezes the split; classifier-run.json contains the actual curve, software and data hashes, and timing; classifier-model.json stores the selected parameters. The recorded initial run took about 3.53 seconds on the stated local CPU setup, including its checks. This is a local observation, not a speed comparison or an expected runtime on other machines.

7.7 Chapter summary

Training requires an update, not just a prediction and a loss. Local rates explain how a small change travels through intermediate values; the chain rule combines them, and backpropagation computes the parameter gradient. An optimizer then applies a declared update. Nonlinear layers expand what can be fitted, including noise. Training, validation and test data answer different questions, and training-set success alone cannot establish generalization.

The loop now works on numerical points. Applying it to language requires another contract first: what exactly is the text that reaches the model? Chapter 8 examines why a screen’s apparently simple string can hide different underlying data.

7.8 Exercises

1. Repeat the starting-point gradient calculation for $x=[1,2]$ and target 0. With step size 0.1, find the new weights, bias and class-1 probability. Which class should become more likely?
2. For target 1 and $x=[2,0]$ at zero weights and bias, compute the parameter gradient and one update of size 0.1. Explain why one weight receives no change from this example.
3. A programmer calls forward and backward for 100 iterations but never calls step(). Another calls step() but never intentionally clears accumulated gradients. Diagnose both cases.
4. At steps 5 and 20, validation accuracy is identical. Which snapshot does the stated rule select, and what difference in evidence justifies it? Explain why training accuracy is not the deciding column.
5. A team uses the test score to choose between ten network widths and calls the winner’s test score an untouched final estimate. Repair the experimental roles.
6. A returned order object passes its JSON schema. Explain why this establishes neither classifier generalization nor permission to cancel the order.

7.9 Reference answers

1. The score derivative is $p-y=1/2$. Multiplying by the input coordinates gives weight gradients [1/2,1] and bias gradient 1/2. New values are w=[−0.05,−0.1], b=−0.05; the score is −0.3 and class-1 probability approximately 0.425557. Class 0 therefore rises to approximately 0.574443, as the target requires.
2. The score derivative is −1/2. The weight gradient is [−1,0], bias gradient −1/2. New values are w=[0.1,0], b=0.05, giving score 0.25. This observation contains $x_1=0$, so varying $w_1$ cannot change its score. Another example can supply a nonzero contribution for that weight.
3. In the first case gradients may be computed but the parameters are never updated. In the second, old gradients can enter later updates unless accumulation was explicitly intended and scaled. The intended loop clears the previous gradients, computes the current loss, backpropagates, then updates.
4. It selects step 5: validation loss 0.581859 is lower than 0.736129, despite both getting 37/48 labels right. Loss also distinguishes the probabilities assigned to targets. Training accuracy measures fit to the data used for gradients, including its four corrupted labels, so it is not the selection criterion.
5. Those test scores have become development evidence. Use a validation split for width selection and reserve another untouched test split for the final chosen procedure. Do not erase the earlier trials from the record. A split’s role depends on how it was used, not what its filename says.
6. The schema checks structure and allowed field values. It does not measure predictions on unseen examples and does not check a user’s authorization for an action. A correctly shaped order_id can still name the wrong order; cancellation needs its own evidence and approval rules.

7.10 References

• PyTorch, automatic differentiation tutorial. Explains computational graphs, backward propagation and gradient accumulation. The chapter’s full one-step arithmetic is independently derived.
• PyTorch 2.7, SGD. Specifies the update mechanism used in the paper example; the nonlinear practice names Adam separately in its fixed configuration.
• PyTorch 2.7, Adam. Documents the moving gradient statistics and defaults used in the measured nonlinear classifier. Its curve is not attributed to the hand-derived SGD rule.
• PyTorch 2.7, CrossEntropyLoss. Documents logits and integer target shapes used by the classifier.
• Zhang et al., Dive into Deep Learning, generalization. Explains why fitting finite observations differs from finding a pattern that carries to new data. The plotted synthetic experiment and all measured values are this book’s own.