Chapter 10 From Sparse Representations to Word2Vec and Neural Language Models

10.1 What could connect hotel with lodging?

In Chapter 9, hotel allowance failed even though the collection contained relevant lodging policies. Keyword, Count and TF-IDF all needed overlap in the document vocabulary. Increasing an IDF cannot create a shared coordinate between unrelated spellings. The failure suggests a different question: can occurrences in similar surroundings teach different words to share useful numerical features?

We should specify what would count as improvement. For retrieval, use the same five KA-0 documents, nine queries, relevance labels and abstention rule. For language prediction, score the same next-token events under each compared language model. A retrieval rank cannot be substituted for a language-model probability, and a lower perplexity cannot by itself establish better retrieval.

This chapter first inspects a deliberately supplied static representation, then explains training signals that can learn representations, and finally follows a small neural language model from IDs to loss. The supplied vectors, actual training run and cited research objectives are separate kinds of evidence.

10.2 Replace vocabulary coordinates with shared features

A BoW vector has one coordinate for each vocabulary term. Another arrangement assigns each word a shorter numerical row, then combines the rows of a document. The same coordinate can contribute to many words; it no longer names one spelling. This is a distributed representation. When most entries are stored as ordinary numerical coordinates rather than a huge set of mostly absent vocabulary terms, it is commonly called a dense embedding. A dense representation can contain zeros; “dense” does not mean every coordinate is nonzero or that the representation understands meaning.

Example 10.1 — A complete hand-assigned diagnostic. To isolate the effect of sharing, give words the following three-coordinate rows. These are authored assumptions, not trained Word2Vec or GloVe output. We deliberately call the coordinates lodging/location, order, and rail for this example only; learned embedding axes do not automatically have those names.

Table 10.1 — All non-default rows in the supplied static map. Unassigned terms, including the two amounts, contribute [0,0,0].

Words	Supplied row
Beijing, lodging, hotel, allowance	[1,0,0]
order, processing, approval	[0,1,0]
train	[0,0,1]
600, 750	[0,0,0]

Represent a card by the arithmetic mean of its token rows, counting zero rows in the denominator. A therefore becomes [2/3,0,0], B becomes [3/4,0,0], C becomes [3/4,0,1/4], and D/E both become [0,1,0]. For hotel allowance, the query mean is [1,0,0]. Cosine against A and B is 1, against C is $3/\sqrt{10}\approx0.948683$, and against D/E is zero. The tie rule chooses A.

That is a successful answer to query 4’s any-version source-finding contract: A or B is relevant. It is not a successful certification of the current 750-yuan amount. Indeed, the same map collapses A and B to the same direction and maps processing and approval to the same row. It fixes one lexical miss while destroying distinctions needed elsewhere. Reducing eight coordinates to three is not evidence of better semantics.

The representation is static because a word’s row stays the same across its occurrences once the map is fixed. A document mean can still change when other words change. What it cannot do is give the individual word a different row according to its current sense. We need training evidence to choose useful rows, and later we will need a way to use context more directly.

10.3 Use neighboring words as a learning signal

The distributional hypothesis motivates a useful kind of evidence: words used in similar linguistic environments often share aspects of meaning or function. Suppose hotel and lodging both occur near room and night. Training that asks their representations to help predict those neighbors may encourage shared features. This is a tendency to exploit, not a theorem of synonymy. Opposites can appear in similar contexts, and a topic association is not equivalence.

Word2Vec names a family of word-representation training architectures. In continuous bag of words (CBOW), surrounding word representations are combined to predict a center word; in skip-gram, a center word helps predict surrounding words. These contexts may include both left and right neighbors. That differs from a left-to-right language model predicting an unseen next token from its available past. Mikolov et al., 2013, §§3.1–3.2

Example 10.2 — See what a skip-gram prediction rewards. Consider a reduced, supplied training situation: the center is hotel, with input row $v=[1,0]$; the only possible context targets are room and train, whose output rows are $u_{\rm room}=[\ln3,0]$ and $u_{\rm train}=[0,0]$. Dot products give scores [ln 3,0]. Softmax gives [3/4,1/4]. If the observed neighbor is room, the loss is $-\ln(3/4)=\ln(4/3)\approx0.287682$.

This training signal favors raising the observed neighbor’s score relative to the alternative. Through Chapter 7’s chain rule, the loss can change the center input row and the context output rows. If another center word repeatedly predicts similar neighbors, the shared prediction task provides a route for statistical sharing. The numbers above demonstrate an objective under supplied values; no actual Word2Vec training is claimed for them.

In full-softmax skip-gram, the general target probability has the form

$$P(o\mid i)=\frac{\exp(u_o\cdot v_i)}{\sum_{j\in\mathcal V}\exp(u_j\cdot v_i)}.$$

$i$ names the center word, $o$ the observed neighbor, and $\mathcal V$ the context-target vocabulary. Input rows $v$ and output rows $u$ play different roles even when the vocabulary spellings are the same. Substituting the two rows above reproduces $3/(3+1)=3/4$. Summing over a large vocabulary is costly. Negative sampling uses a different contrastive objective, distinguishing observed pairs from selected noise pairs; it should not be read as a normalized next-token distribution or used directly to report language-model perplexity. Mikolov et al., 2013, §2

GloVe supplies another learning signal. It first collects word–context co-occurrence counts, then fits dot products and biases to logarithms of those counts. With an authored count of 4 for hotel–room and 1 for hotel–train, the corresponding fitting targets are $\ln4$ and $\ln1=0$, not probabilities 4 and 1. Its objective has the form

$$J=\sum_{i,j:X_{ij}>0}f(X_{ij})\big(v_i\cdot u_j+b_i+\tilde b_j-\ln X_{ij}\big)^2.$$

$X_{ij}$ is a co-occurrence count under a declared context rule; the vectors and biases are fitted parameters. The expression in parentheses is a prediction error in log-count units, squared to penalize either sign. $f$ weights count ranges rather than allowing every count to dominate equally. The zero-count terms are omitted before taking logarithms, consistent with zero weight at zero; we do not calculate log(0) and then multiply infinity by zero. This explains how global co-occurrence evidence trains static rows, without reproducing the paper’s large-corpus experiment. Pennington et al., 2014, §3

10.4 Track a neural language model through one whole prediction

A neural language model (NNLM) can instead learn embeddings jointly with next-token prediction. Replace Chapter 4’s separate context-count rows with shared numerical operations: look up the last few tokens, combine them, compute hidden values, produce logits and score the known next token. Bengio and colleagues’ model establishes this fixed-window, jointly learned representation approach. Our following network is an original small simplification, not a reproduction of their empirical settings. Bengio et al., 2003, §2

Example 10.3 — A fully supplied two-token-window prediction. Use vocabulary order beijing, lodging, 750, EOS, IDs 0–3. The context is beijing lodging, with IDs [0,1]; the supplied next-token target is ID 2, 750. This four-entry paper vocabulary is separate from Chapter 3’s tokenizer and from the measured run below. All values are hand assigned, dimensionless parameters.

The lookup table is

$$E=\begin{bmatrix}1&0\\0&1\\1&1\\0&0\end{bmatrix}.$$

Each of the four token types has two coordinates. For one example, IDs have shape [B,W]=[1,2], where $W=2$ is the window length. Lookup produces rows [[1,0],[0,1]] with shape [B,W,C]=[1,2,2]. Concatenate them in position order to get $x=[1,0,0,1]$, shape [1,4]. Concatenation appends coordinates; it neither adds the vectors nor adds the IDs.

Supply a hidden-layer matrix and bias:

$$H=\begin{bmatrix}1&0\\0&0\\0&0\\0&1\end{bmatrix},\qquad b=[0,0].$$

$H$ has shape [W×C,J]=[4,2], with $J=2$ hidden coordinates. The first output is $1(1)+0(0)+0(0)+1(0)=1$; the second is $1(0)+0(0)+0(0)+1(1)=1$. Adding the zero bias and applying ReLU leaves $h=[1,1]$, shape [1,2].

Finally supply

$$U=\begin{bmatrix}0&0&\ln3&0\\0&0&0&0\end{bmatrix},\qquad d=[0,0,0,0].$$

$U$ has shape [J,V]=[2,4] and $d$ has one entry per output candidate. The logits are $z=hU+d=[0,0,\ln3,0]$, shape [1,4]. Their exponentials are [1,1,3,1], total 6. The target probability is $3/6=1/2$, and target loss is $\ln2\approx0.693147$. Every matrix, bias, input ID and target used in that computation has now been specified.

Lookup and combine

1. Row numberbeijing, lodging → IDs [0,1], shape [1,2]
2. Stored model numbersE[0]=[1,0]; E[1]=[0,1] → [1,2,2]
3. A list of numbersConcatenate → [1,0,0,1], shape [1,4]

Reversing the context changes this concatenation, even though its average would be unchanged.

Predict and score

1. An operationReLU(xH+b) → [1,1], shape [1,2]
2. A list of numbershU+d → [0,0,ln 3,0], shape [1,4]
3. An operationTarget ID 2: p=3/6=1/2; loss=ln 2

The full model can update E, H, U and biases through the same loss. The paper values are supplied, not learned.

Figure 10.1 — A fixed window preserves the order of its two rows by concatenation. Original hand-assigned snapshot, CC BY-SA 4.0; separate from the measured trained model.

Figure 10.1 preserves the distinction between a position axis and a coordinate axis before flattening. The general path is $h=\operatorname{ReLU}(xH+b)$ followed by $z=hU+d$. Backpropagation can update $E,H,U,b,d$ together. The lookup selects parameter rows; it does not differentiate a token’s integer ID. A shared row receives contributions from every occurrence that uses it.

Reverse the context to lodging beijing. Lookup now concatenates [0,1,1,0]; multiplying by this supplied $H$ gives [0,0], all logits become zero and the target probability becomes $1/4$. The average of the two input rows would have remained [1/2,1/2]. Concatenation can preserve order inside the window, while mean pooling cannot.

10.5 Run the small model, keeping its task separate from retrieval

The actual CPU experiment uses the five KA-0 index cards as five training sequences of canonical word symbols. The locale tokenizers map their authored translations to these symbols; this is not evidence of learning English–Chinese translation. Each sequence begins with two input-only BOS markers and ends with one scored EOS. There are 20 training events. No context crosses a document boundary.

The output vocabulary is EOS, beijing, lodging, 600, 750, train, order, processing, approval, IDs 0–8. The input table adds BOS as ID 9. Thus the trained table has shape [10,4], distinct from Example 10.3’s [4,2] table. With a batch of 20 events, the measured network follows [20,2] → [20,2,4] → [20,8] → [20,8] → [20,9]: IDs, embeddings, concatenation, hidden values, logits. Targets have shape [20]. BOS is never an output candidate; EOS is scored. The table contributes $10\times4=40$ parameters, the hidden layer $8\times8+8=72$, and the output layer $8\times9+9=81$, for $40+72+81=193$ trainable numbers.

Table 10.2 — Frozen language-prediction splits. Each displayed sequence also scores its final EOS. These short variants are authored diagnostics, not a representative language corpus.

Split	Sequences	Scored events
Train	The five complete index cards in Table 9.1	20
Validation	beijing lodging 750; order order processing	8
Test	beijing lodging lodging 600; beijing train	8

We use seed 23, CPU float64, one thread, embedding dimension 4, hidden width 8, ReLU, and Adam at learning rate 0.03 for 200 full-batch updates. The first minimum of validation mean loss selects step 10: train loss 1.258041, validation loss 1.454214. The two test sequences are scored only after that selection. This is a completed local run; the record includes all intermediate losses, model values, hashes and elapsed time. No pretrained weights or model account is used.

Table 10.3 — Every held-out target, with two comparable language models. The count baseline is a token trigram LM with add-one smoothing over the same nine outputs, fitted to exactly the same five training sequences. Repeated contexts in different sequences remain separate scored events.

Context	Target	Count target probability	NNLM target probability	Count top choice	NNLM top choice
BOS, BOS	beijing	$2/7$	0.626829	beijing	beijing
BOS, beijing	lodging	$1/4$	0.206176	lodging	beijing
beijing, lodging	lodging	$2/11$	0.198903	lodging	lodging
lodging, lodging	600	$1/10$	0.050457	750	EOS
lodging, 600	EOS	$1/5$	0.735836	EOS	EOS
BOS, BOS	beijing	$2/7$	0.626829	beijing	beijing
BOS, beijing	train	$1/12$	0.044952	lodging	beijing
beijing, train	EOS	$1/5$	0.418215	EOS	EOS

Check one count fraction independently. In training, context beijing lodging is followed once by 600 and once by lodging. Add-one smoothing gives the target lodging numerator 2 and denominator $2+9=11$. For lodging lodging, only 750 was observed, so an unseen 600 receives $1/(1+9)=1/10$. The shared fixture preserves every count fraction before the neural code runs.

Average the eight negative natural logs, including EOS, as in Chapter 6. The count LM gives 1.700367 nats, perplexity 5.475957, and 6/8 correct top choices. The NNLM gives 1.424429 nats, perplexity 4.155483, and 5/8 correct top choices. Its better average likelihood coexists with fewer correct top choices: it puts much more probability on some observed targets, including several EOS events, while missing others. Neither result establishes better current-policy answering. These are language-prediction metrics over eight particular events.

10.6 Reuse learned rows on the unchanged retrieval task

We can take the trained NNLM’s input rows, average them over each KA-0 card or query, and evaluate cosine retrieval. This is a new use of the learned parameters; the NNLM was optimized for next-token loss, not document relevance. Unknown terms receive zero rows in this diagnostic. Mean pooling discards the order that the NNLM’s concatenation used during language prediction.

Table 10.4 — The eight trained lexical rows used for retrieval. Values are from the selected step-10 snapshot, displayed to six decimals. The full-precision file is shared; none of these axes is assigned a semantic name.

Term	Coordinate 0	Coordinate 1	Coordinate 2	Coordinate 3
beijing	0.569753	−0.722903	−0.734649	0.059505
lodging	−0.795324	0.581996	−1.227129	0.518582
600	−2.142605	0.133056	0.276963	2.535696
750	−0.374394	1.196375	1.021346	0.727679
train	1.584991	0.127829	−0.787009	−0.355882
order	0.718588	0.659329	−0.136400	−0.680204
processing	1.036222	1.458809	−0.177986	0.490404
approval	1.488073	0.489607	0.885100	1.945827

For query 1, average the beijing and lodging rows; for B, average beijing, lodging, lodging and 750. Their cosine is 0.734791. Against A it is 0.642097, so B ranks first. These numbers follow from the learned rows and the declared pooling rule; they do not identify a coordinate meaning “current.”

Table 10.5 — Retrieval on all the original queries. Count and TF-IDF have eight coordinates; the supplied static map has three and the learned mean four. The new columns give complete rankings, with first position as prediction. A dash is abstention; its stored all-zero order is A–E. Earlier lexical rankings remain in Table 9.7.

Query	Count first	TF-IDF first	Supplied static ranking	NNLM-row mean ranking
1, current lodging	B	B	A > B > C > D > E	B > A > C > D > E
2, geographic status	C	D	A > B > D > E > C	D > E > C > B > A
3, explicit 750	B	B	A > B > C > D > E	B > A > D > E > C
4, hotel allowance	—	—	A > B > C > D > E	—
5, historical 600	A	A	A > B > C > D > E	A > B > E > C > D
6, processing	D	D	D > E > A > B > C	D > E > C > B > A
7, approval	E	E	D > E > A > B > C	E > D > C > B > A
8, obsolete premise	A	A	A > B > C > D > E	A > B > C > E > D
9, out of scope	—	—	—	—
Successful cases	6/9	7/9	4/9	7/9

The supplied map fixes the synonym query because we explicitly gave its words a shared direction. It also confuses approval with processing and cannot separate the policy versions by direction. The learned mean happens to match TF-IDF’s top-choice success count on this tiny collection, although lower ranks differ. It still fails the synonym query: neither hotel nor allowance occurs in its training vocabulary. Being a neural model does not provide an unlearned word row for free.

Query 8 remains a failure under every representation. Its known terms exactly reproduce the archived card’s direction, producing cosine 1 there. Version and current-intent handling remain missing capabilities. Preserve the lexical outputs alongside the new ones in the continuing KA project rather than replacing the old baselines after one favorable comparison.

10.7 A static row merges senses; a window can lose the clue entirely

The English word train provides a simple ambiguity: it can refer to a rail vehicle or to fitting a model. Our actual corpus uses only its rail sense. A single frozen row cannot switch senses when the same spelling appears in train a model. Other words may change a sentence’s mean, but the train row itself is unchanged. Word2Vec and GloVe also assign static rows in their basic forms; training on a larger corpus can mix senses in that row rather than selecting one per occurrence.

A fixed-window NNLM has a different limitation. It can use different combinations of words inside its window, but a clue outside that window is absent from its input. Increasing hidden width or training longer cannot recover information that the input mapping has made identical.

Example 10.4 — Prove a window collision without blaming unknown words. Use a separate declared vocabulary red, blue, filler, key, is, IDs 0–4. A diagnostic prompt states the key’s color at the start, repeats six identical filler tokens, then asks with the final words key is. The required output color is the one stated at the start. The two ID sequences are [0,2,2,2,2,2,2,3,4] and [1,2,2,2,2,2,2,3,4]; targets are red and blue respectively.

With window length 2, both inputs become [3,4]. Fixed parameters therefore produce the same output distribution for both, whatever the hidden width. This is a representational impossibility under the stated interface, not a measured failure of our KA vocabulary on unfamiliar words. For a deterministic top-color rule, the model cannot answer both correctly. Randomly choosing different outputs does not make the choices depend on the discarded clue.

Red clue

1. Text piecesred, filler × 6, key, is
2. Row numberVisible final window: [3,4]
3. Application recordRequired color: red (0)

The initial color has already left the window.

Blue clue

1. Text piecesblue, filler × 6, key, is
2. Row numberVisible final window: [3,4]
3. Application recordRequired color: blue (1)

Same input IDs and fixed parameters imply the same output distribution.

Figure 10.2 — Once the clue is outside the input window, no parameter choice can distinguish these two inputs. Original impossibility example, CC BY-SA 4.0; a separate declared vocabulary, not an observed language-model accuracy result.

Figure 10.2 places the two different clues outside the identical final windows. A larger window postpones the boundary and enlarges the concatenated input and its following matrix. It still has a finite boundary. The motivating question for Chapter 11 is now specific: can we carry a state forward as tokens arrive, so earlier information need not be supplied as a fixed-length concatenation? That is where recurrence belongs; this chapter does not assume it has already been built.

10.8 Practice record and chapter summary

05-representations-and-nnlm.ipynb reproduces the supplied paper calculation, trains the fixed CPU model, verifies its held-out count fractions, and compares both static methods on unchanged KA-0 queries. Shared inputs are nnlm-paper.json, nnlm-data.json and ka-static-v1.json; actual outputs and selected parameters are in nnlm-run.json and nnlm-model.json. The companion README records versions, seeds, candidate orders, precision and execution commands. The original static map is clearly separated from the trained snapshot.

Sparse lexical methods assign coordinates to spellings. Learned embeddings can share features across words because a training objective connects their occurrences, but usefulness depends on data and task. Word2Vec neighbor prediction and GloVe co-occurrence fitting are representation objectives; a fixed-window NNLM predicts a next token and learns its input rows jointly. Mean-pooled retrieval and sequential prediction remain different computations with different metrics. Static senses and missing earlier context are concrete limits, not reasons to declare every dense representation better.

10.9 Part II checkpoint: trace a supported answer

Before moving to recurrence, check the whole chain against the running examples. The following answers are visible so the checkpoint can also serve as a compact diagnostic guide.

Question	Evidence you should be able to reconstruct
What does [2,2,2] mean in Chapter 5?	Two samples, two stored positions each, two coordinates per position; an axis exchange can keep this shape while changing contents
How do two target probabilities become one score?	$1/2$ and $1/4$ give total loss $\ln8$, mean $\ln8/2$ over two events, and perplexity $\sqrt8$
What actually changes a parameter?	For input [1,2], target 1 and zero parameters, gradient descent with step 0.1 changes weights to [0.05,0.1] and bias to 0.05
Why is 24/24 training success insufficient?	The fixed classifier’s final validation loss is 8.150310, versus 0.581859 at the validation-selected step 5
What must text processing preserve?	Raw source and version, needed amounts/negation, and a position mapping when extracting raw spans; a search key alone cannot preserve all of them
Which baseline improvements are established?	Count fixes query 1’s keyword tie; TF-IDF fixes query 2’s geographic distraction; all lexical methods still miss query 4 and fail query 8
What remains after adding learned rows?	The NNLM mean still misses the absent synonyms and follows the obsolete premise; a two-token window cannot distinguish the two color clues above

This chain reaches a useful first knowledge assistant and a complete small training loop. It does not yet provide a representation that uses clues beyond the fixed window, a source-version decision rule or an authorized order-action workflow. Those missing capabilities remain explicit handoffs to later chapters.

10.10 Exercises

1. In the supplied static map, calculate the means and cosine for order processing and order approval. Which lost distinction explains the approval query’s failure? Would renaming the three axes repair it?
2. In Example 10.2, replace the observed neighbor room with train while keeping the same logits. Compute the new loss. Explain why that observation asks for a different update direction.
3. In Example 10.3, use the reversed context and target 750. Compute hidden values, logits, target probability and loss. For a batch of three such examples, give the ID, embedding, concatenation and logit shapes.
4. A colleague points to lower NNLM perplexity and says retrieval improved over TF-IDF. Use both measured tasks to assess the claim, including the number of observations.
5. For the red/blue diagnostic, would adding 100 hidden coordinates solve the failure with window 2? What change in the input interface would be necessary?
6. A proposed Word2Vec report calls a hand-supplied vector plot a trained result and computes perplexity directly from negative-sampling pair scores. Identify the two evidence errors and what a valid comparison would need.
7. Before answering today’s lodging question with 600, a system notes that the retrieved card has cosine 1. Give the next evidence checks, distinguishing similarity, source applicability and permission to act.

10.11 Reference answers

1. Both means are [0,1,0], so their cosine is 1. The map assigned processing and approval the same row and the averaging cannot distinguish them. Renaming axes does not change any value or restore lost information; a different representation or task rule needs to retain the distinction.
2. Train has probability $1/4$, so its loss is $\ln4\approx1.386294$, compared with $\ln(4/3)$ for room. The observed target is now the low-scoring candidate. Its score should rise relative to room’s; the same input vector can receive conflicting contributions from different observed neighbors, which training must combine.
3. Concatenation is [0,1,1,0]; the supplied hidden matrix gives [0,0], all four logits are zero, target probability is $1/4$ and loss is $\ln4$. For batch size 3, the shapes are [3,2], [3,2,2], [3,4], [3,4]. The two final shapes happen to match, but one axis names concatenated input coordinates and the other output candidates.
4. The likelihood comparison is against a count language model on eight held-out target events: NNLM perplexity is 4.155483 versus 5.475957, while top-choice counts are 5/8 versus 6/8. Retrieval is a separate nine-query comparison: both NNLM-row means and TF-IDF succeed on 7/9. Neither number supports the claimed retrieval improvement, and neither tiny diagnostic establishes a population-wide result.
5. No. Both visible contexts remain [3,4], so any deterministic network with fixed parameters receives identical input. It must receive a larger context containing the clue or a carried representation that preserves relevant earlier information. More hidden coordinates alone cannot reconstruct an input difference that was removed.
6. A supplied plot demonstrates only relationships under authored values; trained claims require an actual objective, data, configuration and run evidence. Negative sampling scores distinguish observed/noise pairs and are not automatically normalized next-token probabilities. A perplexity comparison requires a proper declared output distribution and exactly matched token events, vocabulary and boundary rules.
7. Cosine 1 establishes matching direction under the chosen representation. Check whether the question asks for current or historical policy, whether the source version is superseded, and which effective-date rule applies. B supports 750 for the declared current date. An amount answer requires that source evidence; an order action separately requires its authorized operation and approval path. A retrieval score supplies neither applicability nor permission.

10.12 References

• Bengio et al., 2003, A Neural Probabilistic Language Model. Section 2 describes fixed-window embedding lookup, composition and jointly trained next-word prediction. Our supplied trace and tiny CPU run are original simplifications with separately declared shapes.
• Mikolov et al., 2013, Efficient Estimation of Word Representations in Vector Space. Sections 3.1–3.2 distinguish CBOW and skip-gram training roles. No analogy benchmark or speed result is transferred to our diagnostic.
• Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality. Section 2 specifies input/output word roles and contrasts full-softmax prediction with negative sampling.
• Pennington, Socher and Manning, 2014, GloVe. Section 3 motivates fitting weighted log co-occurrence counts. We use the objective to explain the learning signal and do not reproduce a pretrained model or the paper’s benchmark scores.