Chapter 24 Scaling: How Model, Data, and Compute Grow Together

Suppose our next experiment has a fixed compute allowance. We could make Mini GPT wider, expose it to more text, or train for more passes over the same text. Each choice changes a different quantity. The step-600 snapshot already warns that more updates can improve training loss while making validation loss worse. We need a resource estimate and an evaluation rule, not simply “make everything larger.”

This chapter uses assigned budgets that can be calculated on paper. No model in its budget table was trained. We will first count work, then explain what measured scaling studies can add to that count, and finally ask whether a training-optimal choice is also economical to serve.

24.1 Count one matrix use before counting a training run

Recall a row multiplied by a matrix from Chapter 5. If a matrix has 100 entries and each entry participates once in the row computation, there are about 100 multiplications and 100 additions. We count a multiplication and an addition as two floating-point operations, even if hardware can combine them in one instruction. Thus the approximate forward work is 200 FLOPs. FLOPs counts operations; FLOP/s counts a rate. A larger rate does not mean a larger model.

Now suppose the matrix weights that dominate a dense model contain 100 million parameters. Applying those weights for one token costs roughly $2\times100{,}000{,}000$ operations. As an estimation convention, backward computation costs about twice that forward work. Forward plus backward therefore gives $3\times2\times100{,}000{,}000=600{,}000{,}000$ operations per training token. Processing two billion training tokens gives $600{,}000{,}000\times2{,}000{,}000{,}000=1.2\times10^{18}$ operations.

This calculation motivates the familiar dense approximation:

$$C_{\text{train}}\approx 6ND.$$

Here $N$ is the counted dense weight parameters, $D$ is the number of token exposures processed during training, and $C_{\text{train}}$ is estimated training work in FLOPs. The six combines the two-operation multiply/add convention with the approximate factor of three for forward plus backward. Neither a parameter nor a token is a byte; the product estimates computation, not storage.

The estimate omits optimizer work, communication, recomputation and some vocabulary-related terms. It also neglects the context-dependent pairwise attention work when dense matrix operations dominate. A learned embedding row lookup does not multiply every embedding-table entry for every token. Kaplan et al.’s definition explicitly excludes embeddings; other reports use different counting conventions. Our table treats $N$ as the effective dominant dense weight count. Applying six times MG-2’s total 13875 scalars would therefore be a rough proxy, not an exact audit of its forward program.

24.2 Put three choices under one ceiling

Example 24.1 — Spend the same estimated training compute three ways. The assigned ceiling is $1.2\times10^{18}$ FLOPs. M denotes one million and B one billion; all quantities use decimal units. Each row is a candidate budget, with data quality, optimization and architecture otherwise unspecified. The question is which rows fit, not which model is best.

Table 24.1 — Equal training products, different allocations. The exposure/parameter column is a ratio of counts, not a recommended rule.

Choice	Parameters $N$	Training exposures $D$	$D/N$	Estimated training FLOPs
A	100M	2B	20	$1.2\times10^{18}$
B	200M	1B	5	$1.2\times10^{18}$
C	50M	4B	80	$1.2\times10^{18}$

For B, doubling 100M to 200M requires halving 2B to 1B if the product is to stay fixed. For C, halving parameters permits twice as many exposures. Each multiplication gives the same ceiling. If B instead kept A’s 2B exposures, the estimate would become $2.4\times10^{18}$, twice the allowance.

If the available unique pool contains 1B tokens, C’s 4B exposures require reuse or additional acquisition. Four passes over that fixed pool consume the same approximate matrix work as four billion unique tokens, but they are not the same information. Chapter 23’s mixture and overlap records remain necessary. We also cannot know from this table whether B has enough training, whether C has enough capacity, or whether any row retains the updated policy.

Suppose we assign, rather than measure, an effective training rate of $10^{13}$ FLOP/s. A’s duration estimate is $1.2\times10^{18}/10^{13}=120{,}000$ seconds. Dividing by 3600 seconds per hour gives about 33.33 hours. At half that effective rate, the time doubles to 66.67 hours. A hardware specification’s peak rate is not automatically the effective rate: precision, kernels, memory traffic, communication and utilization intervene. This is a sensitivity calculation, not a promise about a GPU.

24.3 A fitted curve can guide a choice, within its conditions

Operation counts do not predict loss. To learn a relationship, researchers train many models while varying parameters and data, record a specified loss, and fit a compact description to those observations. A scaling law in this setting is an empirical regularity with a measured domain and fitting assumptions. The name does not turn it into a natural law.

Example 24.2 — Read a power relationship without mistaking it for a measurement. Let an author supply losses 4, 3 and 2.5 nats at parameter counts 100M, 400M and 1600M, with other limitations assumed absent. The assigned limiting floor is 2 nats. Subtracting the floor leaves excess losses 2, 1 and 0.5. Multiplying parameter count by four halves that excess each time.

Table 24.2 — An assigned curve whose arithmetic is inspectable. These points are a teaching construction, not results from the models in Table 24.1.

Parameters	Relative size $x=N/(100\text{M})$	Assigned loss	Excess over 2
100M	1	4	2
400M	4	3	1
1600M	16	2.5	0.5

The square roots of 1, 4 and 16 are 1, 2 and 4. Dividing 2 by those roots gives the final column, so the rule is:

$$L(x)=2+2x^{-1/2}.$$

$L$ is the assigned loss in nats per target; $x$ is dimensionless because it divides a parameter count by another parameter count. The exponent $-1/2$ means reciprocal square root. Substituting $x=4$ gives $2+2/2=3$, exactly the middle row. It predicts 2.25 at $x=64$, but that fourth value is an extrapolation outside the supplied range, not another observed point.

On ordinary axes, equal horizontal distances represent equal added parameters. On logarithmic axes, equal distances represent equal multiplication factors. A log-log plot of excess loss, not total loss, would put our repeated fourfold/half changes on a straight line: the slope is $\ln(1/2)/\ln(4)=-1/2$. Chapter 6 introduced logarithms; here their role is to turn ratios into differences. Without subtracting the assumed floor, total loss does not follow that exact straight line. In real fitting the floor and exponent are estimated, not known truths.

The same caution applies when jointly fitting parameter and data limits. One possible empirical form is:

$$\widehat L(N,D)=E+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}.$$

The hat marks a prediction. $E$ is a fitted floor; $A,B$ set the scale in the chosen count units; positive exponents $\alpha,\beta$ describe how the two excess terms fall. None is an accuracy guarantee. For example, with the first excess term fixed at 1 and the second at 0.5, the estimate is $E+1.5$; increasing only data can shrink the second term but leaves the first. A different tokenizer or evaluation distribution changes the meaning of loss and may invalidate the fit.

24.4 Training-optimal means optimal for a stated objective

Under the budget constraint, compute-optimal training asks which tested or fitted allocation gives the lowest specified loss for the available training work. It does not ask which configuration answers a current policy question correctly, has the shortest delay, or is easiest to maintain.

Kaplan et al. studied scaling under their model, data and training setup. Hoffmann et al. revisited parameter/token allocation using several fitting approaches and training horizons, finding roughly joint growth of model size and seen tokens in their experiments. These are empirical studies with different methods. A commonly repeated tokens-per-parameter ratio is not a rule to impose on every dataset, sparse model or deployment. In our table, the ratio 20 identifies A’s arithmetic; it cannot certify A as optimal without a comparable loss experiment.

Fixed training budget

1. Outside the modelA: N = 100M; D = 2B
2. Outside the modelB: N = 200M; D = 1B
3. Outside the modelC: N = 50M; D = 4B

Each candidate has 6ND = 1.2 × 10¹⁸ FLOPs; the product supplies no quality result.

Later generation

1. An operationA: add 2 × 10¹⁸ FLOPs
2. An operationB: add 4 × 10¹⁸ FLOPs
3. An operationC: add 1 × 10¹⁸ FLOPs

Assigned approximation: 2NS, S = 10B output tokens; attention, prompt work and utilization remain outside it.

Figure 24.1 — Equal products do not imply equal quality or serving cost. Original book diagram · CC BY-SA 4.0.

Figure 24.1 adds a second stage to the budget. Suppose all three candidates later generate an assigned $S=10\text{B}$ output tokens, or ten billion. Ignore prompt work and attention/cache effects for this coarse comparison. Forward-only dense work is about $2N$ per output token, so lifetime work becomes:

$$C_{\text{life}}\approx 6ND+2NS.$$

$S$ counts served output-token exposures; it is separate from training exposures $D$. A adds $2\times100\text{M}\times10\text{B}=2\times10^{18}$ FLOPs to training, for a total $3.2\times10^{18}$. B totals $5.2\times10^{18}$; C totals $2.2\times10^{18}$. Equal training products do not imply equal later cost.

This does not prove that C should be deployed: it may fail the quality requirement. It shows why a smaller model trained longer can deserve investigation when later usage is large. Actual lifecycle cost also includes prompt processing, memory capacity, effective execution rate, idle capacity and operations. Chapters 30–33 will attach explicit measurements and an application contract to those terms.

24.5 Summary

Parameters, unique content, processed tokens, FLOPs, bytes and seconds are different quantities. A dense work approximation makes assumptions visible and allows budget comparisons. Empirical scaling fits add evidence about a particular loss under a particular training setup; extrapolation remains a prediction. The model that minimizes training loss at fixed compute need not minimize the cost of an acceptable service. Next we inspect three component choices inside the dense block that these estimates treated as fixed.

24.6 Exercises

1. An assigned dense model has 150M counted parameters and processes 2B tokens. Estimate its training FLOPs. Under the original ceiling, how many tokens could it process?
2. A team reports 4B training tokens from four passes over a 1B-token pool. Which count belongs in the compute formula? What must remain separate in the Data Card?
3. In Example 24.2, calculate the prediction at $x=64$. Explain why plotting that point does not expand the observed range.
4. A model has a lower fitted pretraining loss but fails every explicit policy update. Can “compute-optimal” overrule that failure for the assistant? State the two objectives.
5. C is evaluated at an assigned effective rate of $5\times10^{12}$ FLOP/s. Estimate training hours. Why is substituting a vendor’s peak rate insufficient evidence?
6. Suppose the lifetime output budget falls from 10B to 1B tokens. Calculate A’s and C’s approximate lifetime work and explain what has changed in their comparison.

24.7 Reference answers

1. $6\times150\text{M}\times2\text{B}=1.8\times10^{18}$ FLOPs. Dividing $1.2\times10^{18}$ by $6\times150\text{M}$ gives about 1.333333B token exposures. Rounding the actual training schedule and omitted costs require headroom.
2. Use 4B processed exposures for work. Retain the 1B unique pool, passes/sampling policy, tokenizer and duplication/split information separately. Reuse consumes compute without creating new source information.
3. $2+2/\sqrt{64}=2.25$. It follows the assigned equation beyond the three supplied sizes; no training or evaluation occurred there. A plotted prediction remains a prediction.
4. No. The training objective is loss under a compute constraint; the application objective requires correctly using the supplied effective policy. A candidate can meet the former and fail the latter. Chapter 32’s hard acceptance conditions decide the assistant question.
5. The time is 240000 seconds, about 66.67 hours. The rate is an assigned effective value. Peak hardware throughput does not account for this workload’s precision, utilization, memory or communication, so it cannot establish actual elapsed time.
6. A totals $1.4\times10^{18}$ FLOPs; C totals $1.3\times10^{18}$. The gap shrinks from $1.0\times10^{18}$ to $0.1\times10^{18}$ because generation work is ten times smaller. Quality is still unspecified, and these are coarse work estimates rather than monetary prices.

24.8 References

• Kaplan et al., Scaling Laws for Neural Language Models. Section 2.1 and Table 1 explain the non-embedding count and FLOP convention; its fitted relations depend on the reported training setup.
• Hoffmann et al., Training Compute-Optimal Large Language Models. Section 3 and Appendix D compare fitting routes for parameter/token allocation; projections are distinguished from trained-model evidence.