Chapter 8 Real Text Is Not a Clean String

8.1 Why can two identical-looking searches disagree?

Suppose a travel FAQ contains café and a reader types what looks like the same word. An exact lookup unexpectedly misses. Before changing the retrieval model, ask a more basic question: did the program receive the same data?

Chapter 3 used an exact, versioned tokenizer and showed how an unknown piece loses spelling information. Real text adds an earlier boundary. The screen displays shapes, a string stores Unicode code points, a file stores encoded bytes, and a tokenizer groups text according to its own rules. “The character count” is incomplete unless we name which of those units we are counting.

Our task here is limited: build a reproducible, case-insensitive search key for small English and Chinese examples. We want selected equivalent spellings and repeated whitespace to match while retaining digits, negation and punctuation. We also want to know what we have lost. This is not a universal cleaning recipe, a translation system or a command interpreter.

8.2 Follow one visible mark down to bytes

Example 8.1 — One accent, two underlying strings. The displayed é can be a single code point U+00E9, or an e (U+0065) followed by a combining acute accent (U+0301). A combining mark modifies the preceding letter’s appearance. In a suitable font both forms normally display as one accented letter, but exact string comparison sees different sequences.

A code point is a numbered position in Unicode’s coding space, written here with U+ and hexadecimal digits. Hexadecimal uses sixteen symbols, 0–9 and A–F; the number is an identifier, not a measurement of the letter. A byte contains eight bits and has 256 possible values. An encoding specifies how code points become bytes. UTF-8 uses one to four bytes for a valid Unicode scalar value; one displayed unit may contain several such values.

Table 8.1 — The units disagree even on four tiny examples. Hex byte pairs are shown with spaces only for readability. The tokenizer column applies the unchanged toy-longest-v1 from Chapter 3, which has none of these pieces and performs no normalization.

Displayed text	Code-point sequence	Code points	UTF-8 bytes in hex	Bytes	Toy token IDs
é	U+00E9	1	C3 A9	2	[0]
é	U+0065 U+0301	2	65 CC 81	3	[0,0]
中	U+4E2D	1	E4 B8 AD	3	[0]
🙂	U+1F642	1	F0 9F 99 82	4	[0]

Count the byte pairs in the second row: 65, CC, 81 are three bytes, although there are two code points and usually one visible letter. Python’s len on that string returns 2, whereas len(text.encode('utf-8')) returns 3. Neither number is a token count defined by some other tokenizer. Our toy tokenizer happens to emit one unknown token per unsupported code point, which is why the first two rows produce different ID lengths. Its decoder cannot recover the accent from unknown IDs.

Unicode’s grapheme cluster rules help approximate a user-perceived text unit such as a letter with its combining marks. Some emoji are also sequences containing several code points. A cursor movement or a displayed selection may follow these cluster boundaries, while a Python index follows code points. Scripts and applications can require tailored boundaries; code-point count is not a universal count of what a reader calls a character. Unicode UAX #29, §3

A tokenizer supplies yet another boundary rule: it may combine several letters into one piece, divide a word, or work with bytes. A hypothetical tokenizer that uses one token per byte would produce two tokens for the first row and three for the second, whereas our toy rule produces one and two. This is a comparison of specified rules, not a claim about an unnamed current model. Full subword-training algorithms belong to Chapter 19.

8.3 Decide which distinctions the search key should erase

Exact matching is a useful baseline because it preserves every code point. Its limitation in the accent example is that two canonically equivalent spellings fail to meet. Unicode normalization supplies standardized ways of representing selected equivalent sequences consistently.

NFC performs canonical decomposition and composition; in our example, both accent forms become U+00E9. NFD uses canonical decomposition and yields the e plus combining accent form instead. These operations concern canonical equivalence, not whether two phrases mean the same thing. They do not translate lodging into hotel, remove a negation or discover the correct policy date.

The compatibility forms NFKC and NFKD additionally merge compatibility distinctions. For example, NFKC turns the circled digit ① into 1 and the fullwidth letter Ａ into A. That can help a search box, but a task about original typography or exact identifiers may need the distinction. Normalization is not generally reversible, and normalized pieces can require normalization again after concatenation. Unicode UAX #15, §§1.1–1.4

Case and whitespace decisions are separate from those four forms. For our case-insensitive lookup, we choose Unicode case folding, which makes caseless comparison possible and can do more than lowercase one character. Straße and STRASSE, for example, both become strasse. Collapsing whitespace makes a tab and a run of spaces act like one separator. Neither operation is automatically suitable for a password, signed text, source code or a case-sensitive order ID.

Table 8.2 — The search-key-v1 contract. It is a pure string-to-string operation, with Python’s Unicode database version recorded separately.

Stage	Declared behavior	What it preserves or loses
Input	Require a string; otherwise raise TypeError	Does not silently convert numbers or missing values to text
Canonical form	Apply NFC	Preserves canonical equivalence; loses original composition choice
Caseless comparison	Apply casefold, then NFC again	Loses case distinctions; recomposes sequences introduced by folding
Whitespace	Python split() with no separator, then join with one ASCII space	Collapses recognized whitespace and removes edge whitespace; loses layout
Output	Return the key; do not change the original record	Does not supply a raw-text offset mapping

The order is part of the version. NFC and whitespace handling are not interchangeable labels for “clean.” Nor do we promise to delete every invisible symbol: U+200B, zero-width space, is not treated as whitespace by this Python operation and remains present. The runtime used here contains Unicode Character Database 15.0.0; the consulted specification is revision 57 for Unicode 17.0.0. The selected, long-assigned examples are tested in the declared runtime, rather than claiming that it implements every newer Unicode feature. Python 3.12: unicodedata

8.4 A pure function has visible inputs, outputs and failures

A pure function returns the same result for the same input under the same declared rules, and does not change external state. It is especially easy to test because a failure can be isolated to a small input. The shared implementation reads no files, makes no network request and changes no order record:

import unicodedata

def search_key(raw):
    if not isinstance(raw, str):
        raise TypeError("raw must be a string")
    composed = unicodedata.normalize("NFC", raw)
    folded = unicodedata.normalize("NFC", composed.casefold())
    return " ".join(folded.split())

isinstance checks the input’s type. normalize chooses the declared Unicode form. split() returns the pieces between recognized whitespace, and join reconnects them with a single ordinary space. No file or application is modified; the caller can retain both raw and the returned key. The Python primer explains functions, strings and exceptions before the practice path reaches this file.

Example 8.2 — Predict the output before running the function. In Table 8.3, \t denotes a tab and \n a line break in the supplied string; these are not literal backslash characters. Each result is an authored expectation checked by the implementation.

Table 8.3 — Bilingual inputs and exact search keys.

Input	Output	Inspect the retained distinction
Beijing\tLODGING\n750	beijing lodging 750	750 is unchanged; layout and letter case are lost
北京\t住宿\n750 元	北京 住宿 750 元	Explicit word separators collapse to single spaces
cafe + U+0301	café	The accent remains; its representation changes
Straße / STRASSE	strasse for either input	Two different raw spellings collide
Ａ-104	ａ-104	Width remains under NFC; case still folds
A-104	a-104	This key cannot replace a case-sensitive identifier
Do NOT cancel A-104.	do not cancel a-104.	Negation and punctuation remain
不要取消 A-104。	不要取消 a-104。	Chinese negation also remains
A tab, a newline and a space	Empty string	No search evidence remains

Keeping negation is a necessary preservation choice, not proof that the later system understands it. The function also leaves 北京住宿 without a space and 北京 住宿 with one space. It does not invent Chinese word boundaries. Chapter 9’s small lexical tokenizer will specify its own dictionary matching rule on top of this key.

A useful invariant is idempotence: applying the declared normalization again should leave the key unchanged. The fixture tests this for every retained case, alongside exact outputs, type errors and collisions. Such tests demonstrate selected behavior; they are not an exhaustive conformance test for every Unicode sequence. The companion text-contract.json records the raw values, expected values and counterexamples, and the practice prints them directly.

8.5 Matching improvements can break extraction

Example 8.3 — An offset changes when one code point disappears. Take raw text consisting of cafe, a combining acute accent, a space, and 750. With zero-based code-point indices, the three digits begin at raw position 6: c is 0, a 1, f 2, e 3, the accent 4, the space 5. After NFC, é occupies one position, so 750 begins at position 5.

If an extractor finds the number at normalized indices [5,8)—including 5 but excluding 8—those indices cannot be applied unchanged to the raw string. The raw slice would be a space followed by 75. Preserve raw text and a mapping between representations if the task must quote original spans. A normalized string alone cannot reconstruct which original spelling or whitespace was present.

Two encodings of an accented letter

1. Text piecesé: U+00E9 → C3 A9 (2 bytes)
2. Text piecese + ◌́: U+0065 U+0301 → 65 CC 81 (3 bytes)

NFC gives U+00E9 for both. It does not recover which original representation was used.

An extraction offset is tied to a representation

1. Application recordRaw: c a f e ◌́ ␠ 7 5 0; 750 starts at 6
2. An operationNFC: c a f é ␠ 7 5 0; 750 starts at 5

Keep raw text and a position mapping if results must quote the original span.

Figure 8.1 — Canonical matching can improve while original offsets stop matching. Original examples and diagram, CC BY-SA 4.0; Unicode conventions referenced in the chapter.

Figure 8.1 shows an improvement and a cost from the same operation. Both accent forms can now match, yet the original positions no longer align. The loss is not an implementation bug in NFC; it is a consequence of changing the representation without retaining a mapping.

An even more aggressive “cleaner” might delete short words and punctuation. Deleting not changes do not cancel to do cancel. Dropping currency symbols or decimal points can change an amount; removing all spaces can merge distinct English words. A cleaner cannot be judged by how much shorter or more uniform its output looks. It must be judged against the task’s preserved distinctions.

8.6 Text ambiguity survives correct decoding

The bytes can be correct and the meaning still ambiguous. In I saw her duck, duck can refer to a bird or to someone lowering her head. Normalizing case, accents and whitespace cannot decide between those readings. In Chinese, 苹果 can refer to fruit in one sentence and a company in another. Context, the requested task and the available evidence determine what matters; assigning one permanent label to the spelling will not always suffice.

Task granularity also changes what must be preserved. Finding a relevant policy document needs a document identifier. Classifying an entire question needs a label. Extracting 750 with its unit needs a precise span and its source version. Generating an answer needs enough evidence to state the applicable limit, not merely a normalized digit sequence. Chapter 9 will compare these output contracts explicitly. A text-processing function should not silently claim to do any of them just because it has returned a string.

For the travel example, retain both 600 and 750, and keep the document version and effective-date metadata outside the search key. Lowercasing does not determine which policy applies. For an order such as fictional A-104, retain the authoritative identifier separately from a caseless search key. This practice observes text only; it neither queries a real order nor performs cancellation.

8.7 Chapter summary

Displayed units, Unicode code points, encoded bytes and tokenizer pieces are distinct layers. Canonical normalization can make selected equivalent strings match; compatibility normalization, case folding and whitespace collapse each erase additional distinctions. A pure, versioned function makes those choices testable. Exact offsets and identifiers require their own preservation rules, and correct text handling does not remove linguistic ambiguity.

We can now state exactly which search text enters a computation. Chapter 9 uses that contract to build three cheap lexical baselines and asks which documents they actually retrieve.

8.8 Exercises

1. For e followed by U+0301, give the code-point count, UTF-8 byte count and the output length in code points after NFC. Explain why none determines the token count of an unspecified model.
2. A service replaces stored order IDs with search_key(order_id). Give a concrete collision or change from this chapter and propose a better storage contract.
3. An extractor returns normalized span [5,8) for the 750 in Example 8.3. What is the correct raw span? What additional information would be needed for arbitrary texts?
4. A proposed improvement changes 不要取消订单 to 取消订单 and normalizes both 600 and 750 to a generic number marker. Could it be acceptable for some task? Explain why it fails current-policy answering and action interpretation.
5. A developer claims that passing the 13 normalization cases proves complete Unicode conformance and semantic equivalence. Separate the evidence actually supplied from the two unsupported claims.

8.9 Reference answers

1. There are two code points and three UTF-8 bytes, 65 CC 81. NFC gives one code point, U+00E9, encoded in two bytes. A tokenizer can use bytes, code points or larger pieces and has its own vocabulary and unknown rule; its version and algorithm are needed to count tokens.
2. A-104 becomes a-104; a case-sensitive system may distinguish them. Straße and STRASSE also demonstrate a many-to-one collision. Preserve the original authoritative ID, store a separate optional search key, and resolve matches back to the original record with ambiguity handling. Never overwrite the identifier with the key merely to simplify matching.
3. The raw span is [6,9). For arbitrary normalization and whitespace changes, retain a position mapping or perform extraction directly against the raw representation. Knowing only that NFC was used is insufficient to recover the original composition or spacing.
4. A task that only groups coarse topics might deliberately ignore some amounts, if its evaluation contract permits that loss. Current-policy answering must distinguish 600 from 750, and action interpretation must distinguish prohibition from instruction. The proposed transformation erases exactly those distinctions; a shorter string is not an improvement under those tasks.
5. The tests show the declared implementation matches selected expected inputs, is idempotent on those cases, and rejects wrong types. Complete Unicode conformance requires much broader specified coverage. Semantic equivalence is a different claim entirely: the function neither understands the sentence nor verifies that lost case, layout or offsets were irrelevant to the user’s task.

8.10 References

• Unicode UAX #15, revision 57. Sections 1.1–1.4 specify canonical and compatibility normalization and their boundaries; the stability provisions explain how assigned examples relate across versions.
• Unicode UAX #29, revision 47. Section 3 explains grapheme-cluster boundaries and their relation to user-perceived characters. Our lexical tokenizer is not an implementation of this segmentation standard.
• Python 3.12, unicodedata. Documents normalization functions and the runtime Unicode database version. The book’s search-key choices and loss examples are original, with exact strings in the shared fixture.