Chapter 6 From Logits to Probability, Loss, and Perplexity

6.1 A score needs a rule before it becomes a probability

Chapter 5 produced a row such as [2,1,−1] for Paris, Lyon, other. The first entry is largest, but calling it a probability would make no sense: 2 exceeds 1 and −1 is negative. Dividing all three scores by their sum does not repair the negative entry. We need a rule that preserves which score is larger while producing nonnegative values summing to one.

Such unrestricted pre-probability scores are called logits. A logit is not an observed frequency, a confidence guarantee or a fact about the world. It is an input to the probability rule. This chapter uses supplied logits so we can inspect that rule separately from the model that might produce them. The units of these teaching logits are dimensionless, and the candidate order always travels with the numbers.

Exponentiation provides positive weights. Before using it, recall that $2^3=2\times2\times2=8$, while $2^0=1$ and $2^{-1}=1/2$. An exponential extends this relationship to noninteger exponents. The standard base here is the number $e\approx2.71828$; $e^z$, also written $\exp(z)$, is positive for every finite real $z$. Its inverse is the natural logarithm, $\ln$: $\ln2$ means the exponent to which $e$ must be raised to obtain 2. Thus $e^{\ln2}=2$ and $\ln1=0$.

Example 6.1 — Turn three supplied logits into one distribution. Let the logits be $[\ln2,0,0]\approx[0.693147,0,0]$, still in Paris, Lyon, other order. They are a new teaching input, not the output of Chapter 5’s particular matrix. Exponentiating gives [2,1,1]. The sum of these weights is $2+1+1=4$, so divide each by 4: the probabilities are [1/2,1/4,1/4].

The denominator is 4 because it sums candidate weights. It is not 3 merely because there are three candidates. The general operation, called softmax, is

$$p_k=\frac{e^{z_k}}{\sum_{j=0}^{K-1}e^{z_j}}.$$

$z_k$ is candidate $k$‘s logit and $p_k$ its resulting probability; $j$ visits every candidate in the same row. $K=3$ here. For Lyon, $k=1$, so $p_1=e^0/(e^{\ln2}+e^0+e^0)=1/4$. The probabilities sum to one because their numerators collectively equal their shared denominator. In a [B,T,K] batch, this normalization is performed separately along the candidate axis at each sample and position. Normalizing over all positions together would answer a different question.

6.2 The same distribution can have very different-looking logits

Only differences between logits affect softmax. Adding the same number to all candidates does not favor one over another. This is also the key to computing probabilities without unnecessarily huge exponentials.

Example 6.2 — Add 1,000 without changing the prediction. Use $[1000+\ln2,1000,1000]$. In exact arithmetic, the weights are $[2e^{1000},e^{1000},e^{1000}]$ and their total is $4e^{1000}$. The common factor cancels, leaving the same [1/2,1/4,1/4]. Directly computing $e^{1000}$ in ordinary floating-point arithmetic, however, overflows its finite range.

Subtract the largest logit $m=\max_j z_j$ first. The shifted values are [0,−ln 2,−ln 2]; their weights are [1,1/2,1/2] and total 2. We again get [1/2,1/4,1/4]:

$$p_k=\frac{e^{z_k-m}}{\sum_j e^{z_j-m}}.$$

Every exponent is now at most zero, so no exponential exceeds 1. Small rounding differences remain; our float64 reproduction agrees within $10^{-12}$ for this example. Very negative differences can still underflow to zero. For computing a target’s loss, we therefore use the directly equivalent log expression developed below rather than first rounding a tiny probability to zero.

Adding 1,000 to only Paris is a different operation: it changes relative scores and therefore the distribution. Neither shared shifting nor numerical stability adds evidence that Paris is the correct answer.

6.3 Measure the error at the target that actually occurred

A probability vector needs a target before it can receive a prediction loss. Suppose the recorded target is Paris. Our distribution gave that event probability $1/2$. A second predictor giving it $1/4$ should receive a larger penalty. A loss is the numerical rule that assigns that penalty; we choose a rule compatible with Chapter 4’s multiplication of successive target probabilities.

If two scored events receive probabilities $1/2$ and $1/4$, their product is $1/8$. A long sequence can make this product extremely small. Logarithms convert multiplication into addition: $\ln(ab)=\ln a+\ln b$. Negating the result gives a nonnegative penalty, since the logarithm of a probability between 0 and 1 is nonpositive. This gives the negative log-likelihood of a target:

$$\ell=-\ln p_y.$$

The subscript $y$ identifies the recorded target, not the largest-probability candidate. For $p_y=1/2$, $\ell=-\ln(1/2)=\ln2\approx0.693147$. For $p_y=1/4$, $\ell=\ln4\approx1.386294$. Multiplying probability by one half adds $\ln2$ to the loss; the penalty thus reflects relative probability changes. Probability 1 gives loss 0; probability approaching zero gives unbounded loss. A target assigned exactly zero has infinite negative log loss.

Example 6.3 — Score two targets, then average. Consider two prediction positions that happen to have the same supplied distribution. The first recorded target is Paris and the second is Lyon. This is an arithmetic diagnostic, not a claim that Lyon is the capital of France.

Table 6.1 — Each loss follows its own target.

Scored position	Target	Target probability	Loss in nats
0	Paris	$1/2$	$\ln2\approx0.693147$
1	Lyon	$1/4$	$\ln4\approx1.386294$
Total	Two events	Product $1/8$	$\ln8\approx2.079442$
Mean	Divide total by 2	—	$(\ln8)/2\approx1.039721$

The average divides by the number of scored target events, here 2. It does not divide by the three candidates. For a batch with padding, count genuine scored targets, not padded storage positions. More generally, with $M$ scored events,

$$L=\frac1M\sum_{i=0}^{M-1}\ell_i.$$

$i$ selects an event, $\ell_i$ is its target loss, and $L$ is the mean. Equal weighting per event gives a longer sentence more total contribution than a shorter one; averaging sentence means instead would be a different weighting policy. EOS inclusion must also be declared. Chapter 4’s sequence calculations include EOS.

There is another useful name for this same loss. Represent a known single target as a distribution $q$: 1 at its candidate and 0 elsewhere. For Paris, $q=[1,0,0]$. The cross-entropy from reference distribution $q$ to prediction $p$ is

$$H(q,p)=-\sum_k q_k\ln p_k.$$

The reference probabilities are weights on the candidate losses. With $q=[1,0,0]$, only $-\ln p_0$ remains, giving the target loss above. If $q$ describes a mixture of outcomes, the expression averages their losses using those mixture weights. A term with $q_k=0$ contributes zero; $q_k>0$ paired with $p_k=0$ makes the cross-entropy infinite.

PyTorch’s CrossEntropyLoss accepts raw logits and handles the softmax/log calculation internally. With integer class-index targets, each target says which candidate to score; it does not contain a probability vector. Feeding already normalized probabilities as though they were logits would normalize them again and change the loss. PyTorch 2.7: CrossEntropyLoss

For a single target, the stable form is

$$\ell=m-z_y+\ln\sum_j e^{z_j-m}.$$

It follows by substituting softmax into $-\ln p_y$ and splitting the logarithm of a quotient. In Example 6.1, $m=\ln2$. For Paris, $m-z_y=0$ and the remaining sum is 2, giving $\ln2$. For logits [0,−1000] and target 1, the formula yields approximately 1,000 even if the directly computed second probability underflows. That numerical zero is a storage limitation, distinct from a mathematical model assigning exact probability zero.

One prediction; candidate order Paris, Lyon, other

1. A list of numbersLogits: [ln 2, 0, 0]
2. An operationExponentials: [2,1,1]; total 4
3. A list of numbersProbabilities: [1/2,1/4,1/4]

Divide each weight by the sum of all three candidate weights.

Two labeled predictions with that same distribution

1. An operationParis target: −ln(1/2) = ln 2
2. An operationLyon target: −ln(1/4) = ln 4
3. A list of numbersMean: (ln 2 + ln 4)/2; PPL: √8

Divide total loss by two scored targets, not by three candidates or padded storage slots.

Figure 6.1 — Candidate normalization and target averaging use different denominators. Original diagram and derived teaching arithmetic, CC BY-SA 4.0.

Figure 6.1 shows why the two denominators differ: the top row distributes weight among alternatives for one event, while the bottom row averages penalties across two observed targets.

6.4 Put average loss back on a probability scale

Natural logarithms measure information in nats. Logarithms to base 2 measure it in bits: $\log_2(1/2)=-1$ and $\log_2(1/4)=-2$. Changing the base changes the unit, not which prediction assigned the target more probability. The conversion is $\log_2 p=\ln p/\ln2$; Example 6.3 has mean loss 1.5 bits instead of approximately 1.039721 nats.

Perplexity reverses the logarithm after averaging:

$$\operatorname{PPL}=e^L =\left(\prod_{i=0}^{M-1}p_{y_i}\right)^{-1/M}.$$

$L$ must be in nats in this formula. In bits, use $2^{L_{\rm bits}}$. The product contains the probability assigned to each actual target; the exponent $1/M$ takes the geometric mean, and the negative sign takes its reciprocal. With probabilities $1/2$ and $1/4$, perplexity is $(1/8)^{-1/2}=\sqrt8\approx2.828427$. It is not the arithmetic mean of the reciprocals, which would be $(2+4)/2=3$.

If a predictor assigned every target probability $1/k$, its perplexity would be $k$. That gives an interpretation as an equivalent uniform difficulty. It does not mean the model literally considered 2.828427 alternatives or was wrong that many times. A bad predictor can have perplexity larger than its vocabulary size if it repeatedly puts very little probability on the actual targets.

Comparisons require the same events and tokenization, the same EOS and padding rules, and the same averaging convention. A word-token perplexity and a byte-token perplexity have different event units. Lower perplexity on a fixed held-out text collection means higher geometric-mean probability on those recorded targets. It is not automatically a better answer, a faster system or a more useful product.

6.5 Compare distributions without turning the comparison into a truth test

Sometimes the reference is a distribution rather than one observed label. We want to know how much additional average log loss comes from using $p$ instead of the reference’s own probabilities $q$. That quantity is the Kullback–Leibler divergence, or KL:

$$D_{\rm KL}(q\|p)=\sum_k q_k\ln\frac{q_k}{p_k} =H(q,p)-H(q),\qquad H(q)=-\sum_k q_k\ln q_k.$$

$H(q)$ is the reference distribution’s own average negative log probability, called its entropy. Subtracting it leaves the additional average loss under $p$. The weights come from the left-hand distribution $q$, so swapping the two arguments generally changes the answer. For valid finite distributions, KL is nonnegative and is zero when the distributions agree; it is not a symmetric distance. The information-theory text derives the cross-entropy decomposition for a fixed reference distribution. Dive into Deep Learning, §22.11

Example 6.4 — Direction changes the weighting. In a separate two-outcome experiment, set $q=[3/4,1/4]$ and $p=[1/2,1/2]$. Then

$$D_{\rm KL}(q\|p)=\tfrac34\ln(3/2)+\tfrac14\ln(1/2)\approx0.130812,$$

whereas

$$D_{\rm KL}(p\|q)=\tfrac12\ln(2/3)+\tfrac12\ln2\approx0.143841.$$

The first computation weighs the two log ratios by 3/4 and 1/4; the second uses 1/2 and 1/2. Individual terms can be negative even though the total is nonnegative. Here $H(q)\approx0.562335$ and $H(q,p)=\ln2\approx0.693147$; their difference reproduces the first KL.

The support of a distribution is the set of outcomes with positive probability. For finite $D_{\rm KL}(q\|p)$, every positive-$q$ outcome must also have positive probability under $p$. If $q=[1,0]$ and $p=[0,1]$, the first outcome alone makes KL infinite. A zero-$q$ term contributes zero, including where both probabilities are zero. Artificially replacing zeros with a small constant changes the distribution and the comparison; it must be an explicit modeling choice, not a hidden repair.

6.6 A low loss can reward the wrong document

Return to the fictional travel policy from Part I. The archived Beijing lodging limit is 600 yuan per night; the applicable update is 750. Imagine an evaluation file still labels 600 as the target. A predictor assigns 600 probability 0.99, giving loss $-\ln0.99\approx0.010050$. It scores well against that file and fails the current-policy task. These are authored conditions, not an observed accuracy result.

Even correct labels do not make a probability a factual certificate. Calibration asks whether, across a declared population, events given a probability such as 0.9 occur about 90% of the time. One prediction at 0.9 can still be wrong. A calibrated forecast over one collection also need not remain calibrated after the task or data changes. Factual correctness asks whether this particular claim is supported by the applicable evidence. The two questions require different checks.

The fixed-logit practice in 02-shapes-and-loss.ipynb reproduces Tables 6.1 and the KL values, tests a common shift of 1,000, and checks exact-zero support and numerical underflow separately. The inputs and symbolic expectations live in data/part-ii/arithmetic.json; code/part-ii/probability_practice.py and PyTorch provide two computational paths. Display rounding is six decimals; comparisons use unrounded values. Nothing in that successful arithmetic validates a policy document’s date.

6.7 Chapter summary

Softmax normalizes positive exponential weights within a candidate row. Negative log loss scores the recorded target; cross-entropy extends it to a reference distribution. Averaging requires a declared event count, and perplexity reverses that mean logarithm using a matching base. KL measures additional average log loss in a specified direction with explicit support conditions. These quantities measure predictions against chosen targets or distributions, not the whole quality of a product.

The missing operation is now precise: changing parameters should change logits so that relevant target loss falls. Chapter 7 follows one such change all the way through a training step.

6.8 Exercises

1. Use logits [0,ln 3] in candidate order old, current. Calculate both probabilities, the loss for current, and the loss after adding −50 to both logits.
2. Three genuine targets have probabilities $1/2,1/2,1/4$; one stored position is padding. Find mean loss in nats and perplexity. What happens if someone divides total loss by four?
3. A report converts mean loss 1.5 bits into perplexity using $e^{1.5}$. Repair the result and explain the unit error.
4. Compute $D_{\rm KL}(q\|p)$ for $q=[1,0]$, $p=[1/2,1/2]$. Compute the reverse direction, applying the zero convention explicitly.
5. A new system reduces perplexity by switching to a different tokenizer and scores an old policy file very well. What evidence is missing before claiming that it answers current travel questions better?
6. With [0,−1000] and target 1, a naive implementation returns infinite loss from −log(0). Is that the same failure as assigning exact probability zero in a count model? Explain a repair.

6.9 Reference answers

1. Weights are [1,3], total 4, giving [1/4,3/4]. The target loss is $-\ln(3/4)=\ln(4/3)\approx0.287682$. A shared shift multiplies both weights by the same factor, so probabilities and loss remain unchanged; subtracting the maximum avoids computing unnecessarily small weights.
2. Total loss is $\ln2+\ln2+\ln4=\ln16$. Divide by three targets: $L=(\ln16)/3\approx0.924196$, so perplexity is $16^{1/3}\approx2.519842$. Dividing by four gives $\ln2$ and perplexity 2, falsely improving the score by treating padding as a free event.
3. Use $2^{1.5}=\sqrt8\approx2.828427$. Alternatively convert to nats, $1.5\ln2$, then exponentiate with base $e$. Reversing a logarithm requires the same base used to take it.
4. The first direction is $1\ln2=\ln2$; the second reference term has weight zero and contributes zero. In the reverse direction, the second outcome has reference weight $1/2$ but proposal probability zero, making KL infinite. The shared candidate order does not make the two directions interchangeable.
5. Perplexity needs a matched event/tokenization convention to be directly comparable. The task claim additionally needs a frozen set of current questions, applicable source versions, relevant labels and observable answer errors. Good likelihood on an outdated target file supports only agreement with those targets.
6. In exact softmax arithmetic the second probability is positive, although too small for the chosen storage path. Compute loss directly as $m-z_y+\ln\sum_j e^{z_j-m}$, giving approximately 1,000. An unsmoothed count model may truly assign zero under its stated rule; that is a modeling support issue, not floating-point underflow.

6.10 References

• PyTorch 2.7, CrossEntropyLoss. Specifies logits, target forms, class axes and reductions used in the practice. Our simple mean is unweighted; weighted and ignored-target cases have their own documented denominators.
• Zhang et al., Dive into Deep Learning, information theory. Sections 22.11.4–5 support the relation among entropy, cross-entropy and directional KL. Our finite distributions, numerical values and policy counterexample are original teaching constructions.
• scikit-learn, probability calibration. The opening definition and section 1.16.1 explain calibration through observed frequencies and distinguish it from a loss summary. This chapter does not claim a measured calibration result.