Chapter 3 How Text Becomes Tokens, IDs, and Embeddings

3.1 The problem: open-ended text, finite model inputs

The generation path in Chapter 2 begins with text and quickly reaches lists of numbers. That transition hides several design choices. Text can contain unfamiliar words, punctuation, many writing systems, and symbols never encountered in a particular training sample. A model nevertheless needs a definite set of input entries and a reproducible way to identify them.

The problem is therefore not merely “How do we assign numbers to words?” It is: how do we represent varied text with a finite vocabulary, preserve the distinctions we need, and connect the resulting identifiers to numerical computation? There are three separate operations: choose pieces, identify those pieces, and retrieve their numerical representations. Combining them into the vague phrase “turn words into vectors” obscures where each design decision and failure occurs.

To see why pieces are useful, consider English cat and cats, the singular and plural forms. A dictionary containing only the whole entry cat cannot identify the entire spelling cats as one entry. But if s is also available, it can represent the spelling as cat followed by s. A finite supply of pieces can thus describe more texts than a list containing only complete words.

Consider two possible starting points. A whole-word vocabulary makes common words compact, but a new spelling may require an unknown entry. A character vocabulary can assemble unfamiliar words from known characters, but usually requires more positions for the same text. It also needs a policy for characters outside its coverage. Subword schemes offer another allocation of the vocabulary: store frequent multi-character pieces while allowing some words to be assembled from smaller pieces.

Table 3.1 — Three illustrative ways to represent the same spelling.

Scheme assumed for this comparison	Entries needed for cats	Encoded sequence	Main issue exposed
Whole-word	cats	[cats]	The whole spelling must have an entry
Character	c, a, t, s	[c][a][t][s]	Four positions instead of one
Subword	cat, s	[cat][s]	The boundary depends on available pieces and the selection rule

This comparison does not establish that the shortest encoding gives the best model. The vocabulary, sequence lengths, coverage, and training procedure interact. Subword work such as Sennrich and colleagues’ study addresses the representation of rare words; here we will examine a completely specified teaching scheme, not reproduce its BPE training algorithm. Sennrich et al., 2016

Our immediate goal is an interface small enough to execute by hand. The vocabulary and two-dimensional vectors below are original, hand-assigned teaching data. They are not a tokenizer or checkpoint from an existing LLM.

3.2 Specify the vocabulary and the encoding rule

We need a shared list before we can carry out that idea. Think of each entry as a reusable text tile. A tile labeled cat can be used more than once; we need not add another vocabulary entry whenever it appears again. This comparison concerns reuse only: the program stores numbers and text, not physical tiles.

A token type is an entry that the encoding scheme can use. A token occurrence is one use of an entry at a position in a sequence. The same type can occur repeatedly. The vocabulary lists types and their IDs. IDs identify entries; their numerical size does not represent a semantic quantity.

Table 3.2 — Complete vocabulary and embedding rows for toy-longest-v1. The symbol ␠ displays an ordinary space. Rows are numbered from zero: 0 is an ordinary label for the first row, not an indication that the row is missing. To orient yourself, find cat in the middle column and read its ID, 16, to the left. For now, use only those two columns. We return to the stored numerical rows after completing the text examples.

ID	Piece	Stored embedding row
0	<unk>	[0,0]
1	The	[1,0]
2	␠capital	[1,1]
3	␠of	[0,1]
4	␠France	[2,1]
5	␠is	[1,−1]
6	␠Paris	[2,2]
7	.	[0,−1]
8	法	[1,0]
9	国	[1,1]
10	的	[0,1]
11	首都	[2,1]
12	是	[1,−1]
13	巴黎	[2,2]
14	。	[0,−1]
15	␠	[0.5,0.5]
16	cat	[1,0]
17	s	[0,1]

The text-matching rule is independent of the embedding column. Starting at the first unread position, choose the longest vocabulary piece that exactly matches the beginning of the remaining text. Emit its ID, consume that piece, and repeat. Stop when no text remains. If no piece matches, advance by one Unicode code point and emit the reserved unknown ID 0. Unicode is a standard for representing text; a code point is one of its numbered elements. For the unmatched letters and symbols used below, each fallback step advances past one visible character. A visible character can comprise several code points in other text, so the precise rule uses code points. These Unicode numbers belong to the text standard, separately from the IDs we assigned to our vocabulary.

The rule is case-sensitive: an uppercase letter does not match its lowercase form. It performs no normalization, meaning that it does not first replace text with a chosen standard form. It also inserts no beginning or ending markers. ID 0 is reserved for fallback; the literal spelling <unk> is not one of the pieces searched during matching. The nonreserved pieces are unique. Two different pieces of equal length cannot both exactly match the same prefix, so this vocabulary needs no tie-breaking rule.

These details make the procedure executable. “Use subwords” alone would not determine an output: we would still need the vocabulary, the boundary-selection algorithm, normalization rules, and treatment of unmatched input. Likewise, “token count” has no unique value until a particular encoding scheme and version are specified.

3.3 Execute the rule on complete examples

3.3.1 A new word from existing pieces

The instructions now give us a repeatable procedure. At each step, keep three things in view: the text not yet read, the piece selected from its beginning, and the IDs written so far.

Example 3.1 — Encoding cats. Under v1 there is no entry for the entire string. The first unread text begins with cat, so the rule consumes that three-letter piece. Only s remains, and it has its own entry.

Step	Unread text	Longest match	Emitted ID	IDs so far
1	cats	cat	16	[16]
2	s	s	17	[16,17]
3	Empty	Stop	—	[16,17]

The result has one ordinary English word, four letters, and two tokens. These counts describe different objects. In particular, tokenization does not require a piece to be a whole word or to have a useful meaning in isolation.

Now define toy-longest-v2 by keeping every v1 entry and adding cats with ID 18 and row [1,1]. Both cat and cats match the first position, so the longest-match rule selects cats. The new encoding is [18]. The rule and input are unchanged; the vocabulary changed.

For the input cat, v2 still selects ID 16, because a longer piece is usable only if it matches the actual remaining text. For catscats, v1 emits [16,17,16,17], whereas v2 emits [18,18]. Repeated occurrences need no new vocabulary entry.

One token instead of two says something about encoding length. It does not, on its own, establish greater understanding, better factual accuracy, or a better model. The representation must also be compatible with the computation that consumes it.

3.3.2 Spaces and writing systems do not define universal boundaries

The short word shows how one word can use several pieces. A sentence raises the reverse question: can a piece include something we normally separate from a word, such as a space? The answer comes from the same matching rule.

Example 3.2 — The recurring English sentence. Apply the same rule to The capital of France is Paris.:

Position	Piece consumed	ID	Unread text after this step
1	The	1	␠capital␠of␠France␠is␠Paris.
2	␠capital	2	␠of␠France␠is␠Paris.
3	␠of	3	␠France␠is␠Paris.
4	␠France	4	␠is␠Paris.
5	␠is	5	␠Paris.
6	␠Paris	6	.
7	.	7	Empty

At the second position, the space-only piece also matches, but ␠capital is longer. The rule therefore keeps that space with the word. No space or punctuation has been discarded. Concatenating the selected pieces reconstructs the original text.

Example 3.3 — The corresponding Chinese sentence. The text 法国的首都是巴黎。 expresses the same fact. It has no spaces marking word boundaries, but the rule still matches pieces against the unread prefix:

Unread text	Piece consumed	ID
法国的首都是巴黎。	法	8
国的首都是巴黎。	国	9
的首都是巴黎。	的	10
首都是巴黎。	首都	11
是巴黎。	是	12
巴黎。	巴黎	13
。	。	14

The two characters in 首都 (“capital”) form one token, as do the two in 巴黎 (“Paris”). Conversely, 法国 (“France”) becomes two tokens. The encoding follows Table 3.2 rather than a universal rule that a token equals a word, a Chinese character, or a space-delimited segment.

Both sentences happen to require seven tokens in this vocabulary. That is a property of the chosen examples and entries, not an English-to-Chinese conversion ratio. SentencePiece demonstrates why tokenization need not assume words have already been separated in raw text. Its algorithms are deferred to Chapter 19. Kudo and Richardson, 2018

3.4 Identify when encoding loses information

So far, every part of the input has found an exact match. Before using the resulting IDs in a model, we should check what happens when that is no longer true. Does the encoding still preserve the distinction between two inputs?

A useful check for an encoding is the round trip: encode an input, decode the resulting IDs, and compare the reconstructed text with the original. The successful round trips above follow from two facts: each selected piece matches exactly, and decoding concatenates those same pieces in the same order.

Example 3.4 — Two inputs become indistinguishable. For cat🙂, the rule first emits ID 16. The smile has no matching entry, so fallback consumes it and emits ID 0. Decoding can retrieve only cat and <unk>. The same happens to cat🚀.

Table 3.3 — An information-losing collision.

Original input	Encoded IDs	Decoded text	Information no longer present in the IDs
cat🙂	[16,0]	cat<unk>	Which unknown symbol occurred
cat🚀	[16,0]	cat<unk>	Which unknown symbol occurred

The decoder receives identical input for two different original strings. It therefore cannot always reconstruct both originals correctly. This is not a weakness that more careful decoding can remove: the distinction is absent from the representation it was given. Retaining the original text elsewhere would preserve it, but would be a different information source.

Our fallback consumes one Unicode code point, a numbered element in the text standard. A visible character can comprise multiple code points; the smiling symbol used here comprises one. Chapter 8 develops the text representation details. A byte is a small unit used to store text in a computer. Tokenizers that cover all such byte values can avoid this particular unknown-symbol collision, and normalization can introduce other round-trip differences. Our example concerns the stated limited scheme, not every tokenizer.

The same input specification also explains case sensitivity. The matches ID 1. Lowercase the has no matching piece at any of its three positions, so it becomes [0,0,0]. No step silently corrects the case. If a preprocessing stage were added to do so, it would be part of a different encoding contract and would need to be recorded.

3.5 Use an ID as an address, not as a measurement

3.5.1 Retrieve a numerical row

We now know which IDs enter the model and when they lose information. Return to the last column of Table 3.2. This is the next stage from Chapter 2: the ID locates stored values, and those values enter the model’s calculation. The embedding column defines a second operation. For each ID, copy the row at that address, preserving the order of token occurrences.

Example 3.5 — Looking up cats under v1.

Text pieces:         [cat]       [s]
Vocabulary IDs:        16        17
Rows read:           [1,0]     [0,1]
Ordered result:     [[1,0], [0,1]]

One word → two pieces → two numerical rows

cats↓cat | s

Piece 1cat

→

Vocabulary address16

→

Read stored row 16[1, 0]

Piece 2s

→

Vocabulary address17

→

Read stored row 17[0, 1]

Keep the two retrieved rows in input order: [[1,0], [0,1]]. Nothing here adds or multiplies the IDs.

Figure 3.1 — Different objects at each step; the parameter table supplies the vectors. Original book diagram · CC BY-SA 4.0.

Read the last line of the example from the inside out. [1,0] is the list for cat; [0,1] is the list for s. The outer brackets keep those two lists in text order. They do not ask us to add the lists together.

Figure 3.1 separates the piece, address, and retrieved values. The outer result has two rows because the encoded text has two tokens. Each row has two entries because we chose embedding dimension two. A dimension here is the number of coordinates in a vector. Neither token count nor ID magnitude determines that dimension.

An embedding lookup reads model parameters; it does not learn them at that moment. In neural language models, numerical representations can be learned jointly with the function assigning sequence probabilities. This allows statistical information to be shared across inputs through their representations and the shared prediction function. Bengio and colleagues provide an early formulation. Our assigned rows illustrate the interface only; for example, the first coordinate has not been defined as “animalness.” Bengio et al., 2003

3.5.2 Test the claim that nearby IDs mean similar things

Suppose someone argues that cat and s must be similar because their IDs, 16 and 17, differ by only one. Think of two library books with adjacent catalog numbers. Renumbering the catalog would not change either book’s contents. Likewise, we can test the claim about IDs by changing labels while preserving everything each label retrieves.

Swap the IDs of cat and The, and swap their corresponding rows. cat now has ID 1, but still retrieves its original [1,0]; The now has ID 16 and still retrieves its own original row. A generator uses the same labels at its output: it produces an ID that must be read back as a text piece. If we renumber only the input, that output could now name the wrong piece. To preserve a complete generator, also change its output-ID mapping and all other parameters arranged by token ID consistently. Every token can then take the same computational role despite its changed number.

The numerical distance from cat to s changes from one to sixteen. The consistent relabeling did not change the intended model behavior. Therefore that ID distance cannot be an intrinsic measure of their semantic relationship. This argument concerns labels; it does not claim that changing only the input IDs is harmless.

For a clearer mismatch, swap cat with the space entry, whose stored row is [0.5,0.5], but leave the old table untouched. The new cat ID now retrieves the space row. The wrong numerical input reaches every later stage. Compatibility between tokenizer, input embedding table, output mapping, and checkpoint is part of the model interface.

The v2 addition raises another compatibility issue. An unchanged v1 model has no row 18 for cats. Providing a new tokenizer alone cannot supply that row or the corresponding output entry. Extending the vocabulary is a change to the encoding/model interface, not a way to obtain an improved model merely by shortening its input.

3.6 Compare representations with a defined geometric operation

3.6.1 Direction and length are different properties

ID differences did not measure similarity, because IDs were only labels. Embedding values have a different role: they are the numbers used in calculation. We can compare these values, provided we specify the property being compared. One common choice is the direction of their vectors. To understand it, first put our two-number lists on a page. Draw a horizontal and a vertical axis crossing at zero, the origin. The first coordinate gives horizontal displacement: positive goes right and negative goes left. The second gives vertical displacement: positive goes up and negative goes down. [1,0] therefore locates a point one unit right; [1,1] locates a point one unit right and one up. Draw a straight arrow from the origin to each point. The second arrow is diagonal; it does not follow a bent path along the two axes. It is longer and points in a different direction.

u · v = 1; |u| = 1; |v| = √2. Opposite pair: u and w.

Figure 3.2 — Shared coordinates distinguish length from direction; the zero vector would remain at the origin. Original book diagram · CC BY-SA 4.0.

Figure 3.2 places the vectors on the same axes. It also shows [-1,0], pointing left. The coordinates define the geometry; they do not assign meanings to the axes. Cosine similarity compares nonzero vectors by direction while removing the effect of their lengths.

Why separate direction from length? [2,2] points along the same diagonal as [1,1], just twice as far. A direction comparison should treat those two arrows alike. Cosine similarity does so by dividing out the lengths. The calculation below shows both the division and the quantities it needs.

Example 3.6 — Calculating the diagonal case. Let $u=[1,0]$ and $v=[1,1]$. First calculate the quantities needed for the ratio.

Quantity	Operation	Result
Dot product	Multiply matching coordinates and add	1 × 1 + 0 × 1 = 1
Length of u	Square coordinates, add, take the positive square root	√(1² + 0²) = 1
Length of v	Apply the same length rule	√(1² + 1²) = √2

The dot product is the first row’s multiplication and addition. Squaring means multiplying a number by itself. For length, the horizontal displacement, vertical displacement, and straight arrow form a right triangle. The Pythagorean theorem says that the squared arrow length equals the sum of the squared displacements. For [1,1], that sum is 1 + 1 = 2; we need the positive number whose square is 2 to get the length itself. The positive square root of 2 is the number whose square is 2, approximately 1.4142.

The letters u and v simply name the two vectors. In the next expression, the fraction bar means division and the symbol ≈ means “approximately equal.” Divide the dot product by the product of the lengths:

$$\begin{aligned} \operatorname{cos}(u,v)&=\frac{1\times1+0\times1}{\sqrt{1^2+0^2}\,\sqrt{1^2+1^2}}\\[4pt] &=\frac{1}{\sqrt2}\approx0.7071. \end{aligned}$$

Here the first arrow points horizontally, so the result is also the diagonal arrow’s horizontal displacement divided by its length: 1 ÷ 1.4142, approximately 0.7071. That is the cosine of the angle between the arrows, which explains the measure’s name in this example. In the general dot-product expression, the numerator reflects both alignment and length; the denominator compensates for length. To see that compensation, replace [1,1] with [2,2]. The numerator doubles from 1 to 2, and the second vector’s length doubles from √2 to 2√2. The ratio stays 1/√2. The arrow has been stretched without changing its direction.

3.6.2 Opposite directions, and a vector with no direction

Stretching an arrow preserved its direction and its cosine. Turning it around should change the comparison.

Example 3.7 — Reversing one arrow. Keep $u=[1,0]$ and set $v=[-1,0]$. The dot product is 1 × (−1) + 0 × 0 = −1. Each length is 1, so:

$$\operatorname{cos}(u,v)=\frac{-1}{1\times1}=-1.$$

For nonzero vectors, cosine ranges from −1 to 1. Opposite directions give −1, perpendicular directions give 0, and the same direction gives 1. Thus cosine 1 does not require equal entries: [1,0] and [2,0] have different lengths but cosine 1.

For $v=[0,0]$, the length is zero and no direction is defined. Cosine would divide by zero, so its value is undefined. Replacing it with zero would conflate “no direction” with “perpendicular directions.” In particular, our unknown embedding has zero length; it cannot participate in this cosine calculation without an additional convention.

Table 3.4 — Fixed comparison results, with first vector [1,0].

Second vector	Dot product	Its length	Cosine	Interpretation
[1,1]	1	√2	≈ 0.7071	Diagonal direction
[2,2]	2	2√2	≈ 0.7071	Same direction as the preceding row
[-1,0]	−1	1	−1	Opposite direction
[0,1]	0	1	0	Perpendicular direction
[0,0]	0	0	Undefined	No direction

Cosine answers a geometric question. It becomes useful for a language task only when the representation and the task make that geometry informative. If we assign “approved” and “rejected” the same vector, their cosine is 1 although the words can express opposite decisions. Learned vectors are constrained by training rather than arbitrary assignment, but high similarity still does not by itself establish synonymy, logical entailment, or truth.

3.7 Distinguish three uses of the word embedding

We can now calculate a geometric comparison, but a vector alone does not tell us what it represents. Before comparing two vectors in a language task, ask where they came from and what the model was trained to represent. The word “embedding” appears in several settings with different answers.

A token’s input embedding is the row retrieved at the model entrance. Under a fixed tokenizer and fixed table, the same ID retrieves the same row in different sentences. This is the object used in Example 3.5.

A contextual representation is a numerical result computed using available surrounding tokens and position. It may differ when the same token appears in a different sentence. There is no contradiction: the fixed lookup supplies an initial value, and later computation uses that value together with other input. Work on contextualized representations studies the usefulness of this additional information. Peters et al., 2018

A retrieval embedding represents a query, passage, or other item for a search task. It may be produced by a different model and training objective. A retrieval index organizes such representations for search. Our input embedding table is neither a document index nor a complete method for computing a sentence’s search representation.

These distinctions also explain why comparing two token rows does not measure every computation a language model can perform. The row is one early representation. Later stages may combine it with context in ways not visible in an isolated cosine score.

3.8 Summary

Text encoding is a specified procedure, not a universal division into words. The vocabulary and rules determine token boundaries, IDs, coverage, and round-trip behavior. Our finite example represented unfamiliar cats using existing pieces, but lost the identity of unknown symbols. Changing the vocabulary changed token counts and imposed a compatibility requirement on the model.

IDs are addresses. Embedding lookup retrieves numerical rows without treating those addresses as semantic measurements. Cosine compares directions of nonzero vectors; it removes scale and says nothing about truth by itself. Input embeddings, contextual representations, and retrieval embeddings are different objects serving different stages or tasks.

We have now specified the entrance to the model, but have not derived its next-token probabilities. Chapter 4 does that with a model built from counts. It deliberately uses characters directly, so that probability estimation can be studied without first implementing a neural embedding model. General vector and matrix computation begins in Chapter 5; tokenizer learning algorithms wait until Chapter 19.

3.9 Exercises

1. Execute the algorithm. Encode cats cats under v1 and v2, displaying the ordinary space explicitly. Give the IDs and token counts. Which entries are token types, and which are repeated occurrences?
2. Case and boundaries. Encode Thecat and thecat under v1. Explain why a lack of spaces does not prevent the first string from using ID 1, and why the second cannot use it.
3. Round-trip impossibility. Suppose a decoder receives [16,0]. Explain why no deterministic decoder using only those IDs can always recover whether the original was cat🙂 or cat🚀.
4. Vocabulary extension. A third version adds cat🙂 as one exact piece. How would that change Example 3.4? What must also change before a neural model can consume its new ID?
5. Relabeling. State all the correspondences that must remain consistent when two vocabulary IDs are exchanged in a full generator. Why does exchanging only tokenizer IDs fail in general?
6. Scale and direction. Compute cosine between [1,0] and [3,3], then between [1,0] and [-3,-3]. Explain which change preserves direction and which reverses it.
7. Zero is not undefined. Compute the dot product and lengths for [1,0] with [0,1], then with [0,0]. Identify the precise step at which the second calculation fails.
8. Meaning and geometry. In Table 3.2, cat and The have the same assigned row. What is their cosine? What conclusion about meaning does that result fail to establish? Would a later contextual representation necessarily stay identical?
9. Interface audit. A colleague says, “We averaged token IDs to obtain a document embedding.” Explain why an ID relabeling provides a counterexample to treating that average as an intrinsic semantic representation. Distinguish this from averaging suitable numerical representations, whose usefulness would require its own justification.

3.10 Solutions

1. v1 gives [cat][s][␠][cat][s], IDs [16,17,15,16,17], five occurrences. v2 gives [cats][␠][cats], IDs [18,15,18], three occurrences. v1 uses types cat, s, and space, with cat and s each occurring twice; v2 uses cats and space, with cats twice.
2. Thecat becomes [1,16]. thecat becomes [0,0,0,16]. Matching requires an exact prefix, not a preceding or following space, so The can match at the beginning of the first string. Lowercase t, h, and e have no matching entries. The rule does not normalize their case.
3. Both encodings give the decoder exactly the same input. A deterministic function must return the same output for that input, whereas the two originals differ. Recovering the distinction requires additional retained information or an encoding that preserves it.
4. Longest-match encoding would select the new complete piece for cat🙂; its exact decoding could recover the smile. cat🚀 would still fall back unless separately covered. The new ID also needs compatible input and output entries and model state; editing the tokenizer alone supplies none of those numerical parameters.
5. Piece-to-ID mapping, ID-to-piece decoding, input embedding rows, output labels/rows, and other token-indexed parameters must be permuted consistently. Otherwise a piece can retrieve another piece’s numerical input or an output ID can decode to the wrong text.
6. For [3,3], the numerator is 3 and the lengths are 1 and 3√2, giving 1/√2. For [-3,-3], the numerator is −3 and the lengths are unchanged, giving −1/√2. Positive scaling preserves direction; multiplying by a negative number reverses it.
7. With [0,1], the dot product is 0 and both lengths are 1, giving 0. With [0,0], the dot product remains 0 but the second length is 0, so division requires 0/0. The dot product itself is defined; cosine is not.
8. The cosine is 1 because both nonzero rows are [1,0]. It does not establish that cat and The mean the same thing; the rows were assigned by hand. Later values can depend on surrounding tokens and positions, so equal initial rows need not imply equal contextual results.
9. A consistent change of ID labels changes their average even when the model’s intended behavior is preserved. That average therefore depends on arbitrary numbering. Averaging suitable embedding vectors is a different numerical operation on representations, but whether it produces a useful document representation depends on their training and the retrieval task.

3.11 References

• Sennrich, Haddow, and Birch, Neural Machine Translation of Rare Words with Subword Units (2016). The paper uses subword units to address rare words in machine translation. It develops the motivation introduced here; our fixed longest-match tokenizer is a separate teaching construction.
• Kudo and Richardson, SentencePiece (2018). This work describes tokenization learned from raw text without requiring words to have been separated in advance. It helps explain why splitting on spaces cannot be a universal account of tokenization.
• Bengio et al., A Neural Probabilistic Language Model (2003). The model learns word vectors together with a function that assigns probabilities to sequences. It connects the lookup operation in this chapter to learned representations; our illustrative vectors remain hand assigned.
• Peters et al., Deep Contextualized Word Representations (2018). The paper computes word representations from the surrounding sentence. Compare this with a fixed embedding-table row to understand why a contextual representation can change even when the token ID stays the same.