Chapter 9 Keywords, Bag of Words, and TF-IDF: Establishing a Sparse Baseline

9.1 Can we find a useful source before asking a model to write?

A reader asks for the Beijing lodging limit. Part I established why an archived answer of 600 yuan can conflict with a newer limit of 750. Before training a language model to answer, we can ask a simpler, observable question: which supplied document should the system retrieve? If a cheap lexical method succeeds, it is a useful baseline. If it fails, the failed query tells us what a more complicated method needs to improve.

This chapter completes the first knowledge-assistant milestone, KA-0. Its output is a ranked list of documents and one top-document prediction, or an explicit abstention. It does not generate a fresh answer or execute an order operation. We freeze the documents, questions, relevance labels and tie rules before comparing methods. The examples are fictional and deliberately small enough to calculate completely; their success counts are not estimates of performance on everyday search traffic.

9.2 Freeze the text that is actually indexed

The collection contains two versions of a lodging policy and three short FAQs. Each record preserves its full explanatory body and version metadata, but this first experiment scores an explicitly supplied index card. It does not silently count every word in the body. The cards are short, authored summaries; their selection is part of the task assumption, not a learned summarization result.

Table 9.1 — The five documents and their complete indexed text. A–E are reader labels in the fixed candidate order. Amounts in A and B are yuan per room per night under the fictional policy.

Card	Source and scope	Complete indexed text
A	Archived v1: 600; effective 2025-01-01, superseded by B	Beijing lodging 600
B	Current v2: 750; effective 2026-01-01	Beijing lodging lodging 750
C	Rail FAQ; a title, tag and summary repeat Beijing; no lodging or status answer	Beijing Beijing Beijing train
D	Status FAQ: processing is a possible status of fictional order A-104; no live lookup	order processing
E	Cancellation FAQ: canceling an order requires approval; no approval granted by retrieval	order approval

B repeats lodging in its heading and summary. C repeats a location in its title, tag and summary. Repeated text is an intentional part of this diagnostic, not something we remove because a baseline dislikes it.

Apply Chapter 8’s search-key-v1 first. The small English tokenizer then takes ASCII letter runs and digit runs, ignoring separators and punctuation. The Chinese counterpart matches the longest known word in a declared teaching dictionary, handles ASCII/digit runs, and otherwise retains a code point unless it is whitespace or punctuation. Thus 北京住宿 can yield the same two vocabulary entries as 北京 住宿; no general Chinese segmentation accuracy is claimed. The two locales are parallel authored collections with aligned columns, not an automatic translation system.

Scanning the cards in A–E order and adding each newly encountered term gives the vocabulary Beijing, lodging, 600, 750, train, order, processing, approval. We then freeze that order. Query terms outside this document vocabulary do not receive new lexical columns. The tokenizer may recognize the spellings hotel and allowance, but recognizing a spelling is different from giving it a trained synonym relationship or a document-vocabulary coordinate.

9.3 Begin by counting distinct matching keywords

Example 9.1 — Execute the simplest lookup. The query is Beijing lodging, interpreted as a current-policy question. The relevant document is B. Count how many distinct query terms appear in each card: A matches both, B matches both, C matches Beijing, and D/E match neither. The scores are [2,2,1,0,0].

This keyword baseline gives one point per shared term, ignoring repetition and order. Ties follow the frozen order A, B, C, D, E, so the ranking is A > B > C > D > E and the prediction is A. It has found the topic but selected an obsolete source. The failure follows directly from the scoring and tie rules; the program has no date preference hidden inside it.

A zero score for every card means there is no lexical evidence. We return no prediction rather than choosing A merely because it is first. The diagnostic output still records a deterministic ordering of zero-score candidates, but that ordering is not a recommendation.

9.4 Keep repetition with a bag of words

Binary presence discarded a difference between A and B: B contains two occurrences of lodging. To preserve that difference, give each vocabulary term one coordinate and count its occurrences. This count vector is a bag-of-words (BoW) representation: it retains which terms occur and how often, but discards their order.

Table 9.2 — The complete document-term count matrix. Rows are documents; columns follow the frozen vocabulary. Counts have units of occurrences, not probabilities.

Card	Beijing	lodging	600	750	train	order	processing	approval
A	1	1	1	0	0	0	0	0
B	1	2	0	1	0	0	0	0
C	3	0	0	0	1	0	0	0
D	0	0	0	0	0	1	1	0
E	0	0	0	0	0	1	0	1

Read B’s second cell: lodging occurs twice in its index card, so the entry is 2. Stacking the five length-8 vectors gives a document-term matrix of shape [5,8]. It has 12 nonzero cells and 28 zeros. Larger vocabularies often make such representations sparse, with most coordinates zero; an implementation can store nonzero indices and values instead of every zero. Sparse does not mean incomplete relative to this declared vocabulary.

The keyword matrix can be recovered by changing every positive count to 1. For B, its keyword row is [1,1,0,1,0,0,0,0]; its count row is [1,2,0,1,0,0,0,0]. Both forget whether a repeated word appeared in a heading or in an explanation.

Example 9.2 — Compare count directions. The query vector is $q=[1,1,0,0,0,0,0,0]$. Against A, the dot product is $1(1)+1(1)=2$; the query length is $\sqrt2$ and A’s length is $\sqrt3$, so the cosine is $2/\sqrt6\approx0.816497$. Against B, the dot product is $1(1)+1(2)=3$, B’s squared length is $1+4+1=6$, and the cosine is $3/\sqrt{12}\approx0.866025$.

We divide by both lengths so that simply doubling every count in a card does not double its similarity. Against C the dot product is 3 and the denominator is $\sqrt2\sqrt{10}$, giving 0.670820. D and E have no overlapping nonzero coordinates and receive zero. The full score row is [0.816497,0.866025,0.670820,0,0]; BoW Count ranks B > A > C > D > E, fixing this query’s top choice.

It fixed the choice because the count direction changed, not because it understood which version is current. For a second query, Beijing processing, the reader wants the meaning of an order’s processing status, with Beijing as context. Count cosine gives C approximately 0.670820 and D only 0.5. Repeated geography has displaced the useful status term. We need to ask how common each term is across documents, not just how often it occurs inside one.

9.5 Weight a term by how widely it occurs

Term frequency (TF) here is simply raw count: $\operatorname{tf}_{d,j}=c_{d,j}$, the number of occurrences of term $j$ in card $d$. This is one explicit TF convention; dividing by document length or using logarithmic counts would define different variants. TF is not a conditional next-token probability.

Document frequency, $\operatorname{df}_j$, counts how many cards contain term $j$ at least once. Beijing occurs five times in the collection but in only three documents, so its document frequency is 3, not 5. A term appearing in many documents often distinguishes them less than a rare term. This is a collection-dependent statistical argument, not a claim that rare words are inherently important. IIR, inverse document frequency

Example 9.3 — Fix the exact inverse-frequency convention. Let $N=5$ be the number of cards. Our inverse document frequency (IDF) is

$$\operatorname{idf}_j=\ln\frac{N+1}{\operatorname{df}_j+1}+1.$$

The numerator counts the collection size plus one; the denominator counts containing documents plus one. The additions inside the ratio define a smoothed convention, and the final +1 keeps a term present everywhere at weight 1 instead of 0. These are modeling choices, not algebra forced by the name TF-IDF. We use natural logarithms and fit the weights on documents only, never on the query relevance labels.

Table 9.3 — All document frequencies and IDF values. For compact later calculations, name the three distinct weights $a,b,c$.

Terms	df	IDF calculation	Weight
Beijing	3	$1+\ln(6/4)$	$a\approx1.405465$
lodging, order	2 each	$1+\ln(6/3)$	$b\approx1.693147$
600, 750, train, processing, approval	1 each	$1+\ln(6/2)$	$c\approx2.098612$

The TF-IDF coordinate multiplies the count by that term’s weight:

$$v_{d,j}=c_{d,j}\operatorname{idf}_j.$$

B’s lodging coordinate becomes $2b\approx3.386294$. C’s Beijing coordinate becomes $3a\approx4.216395$. A missing term stays zero because its count is zero. Unknown query terms are ignored under the frozen vocabulary rather than inventing a new coordinate with an arbitrary IDF. The maintained scikit-learn documentation describes this raw-count, smoothed-IDF convention; other published formulas, including the classic unshifted $\log(N/\operatorname{df})$, should not be mixed with these numerical results. scikit-learn: TF-IDF weighting

Table 9.4 — Complete weighted vectors before length normalization. Every symbol refers to Table 9.3, not to an unspecified learned value.

Card/query	Beijing	lodging	600	750	train	order	processing	approval
A	$a$	$b$	$c$	0	0	0	0	0
B	$a$	$2b$	0	$c$	0	0	0	0
C	$3a$	0	0	0	$c$	0	0	0
D	0	0	0	0	0	$b$	$c$	0
E	0	0	0	0	0	$b$	0	$c$
Beijing lodging	$a$	$b$	0	0	0	0	0	0

The query uses the same IDF as the documents. Its length is $\sqrt{a^2+b^2}\approx2.200473$. We now complete every document comparison using Chapter 5’s cosine:

$$\operatorname{score}(q,d)=\frac{\sum_j v_{q,j}v_{d,j}}{\sqrt{\sum_j v_{q,j}^2}\sqrt{\sum_j v_{d,j}^2}}.$$

Here $j$ visits vocabulary terms, not text positions; $q$ and $d$ name the query and one document. The denominator removes the lengths of these weighted vectors. It does not divide by word count, document frequency or the number of candidates.

Table 9.5 — A complete TF-IDF query against every card. The numerator uses squared weights because the query and document both carry IDF.

Card	Dot-product numerator	Document length	Full denominator	Cosine
A	$a^2+b^2\approx4.842080$	$\sqrt{a^2+b^2+c^2}\approx3.040765$	6.691120	0.723658
B	$a^2+2b^2\approx7.708827$	$\sqrt{a^2+4b^2+c^2}\approx4.224511$	9.295921	0.829270
C	$3a^2\approx5.925997$	$\sqrt{9a^2+c^2}\approx4.709794$	10.363773	0.571799
D	0	$\sqrt{b^2+c^2}\approx2.696465$	5.933496	0
E	0	$\sqrt{b^2+c^2}\approx2.696465$	5.933496	0

For B, divide $7.708827$ by $2.200473\times4.224511$ using unrounded values, giving 0.829270. D and E have nonzero lengths but zero dot products. A query with no known terms instead has zero length; the application treats that undefined cosine as zero retrieval evidence and abstains. It does not change the mathematical definition of cosine.

The ranking for this first query remains B > A > C > D > E. The improvement appears on Beijing processing: its weighted query has entries $a$ for Beijing and $c$ for processing. Against C the numerator is $3a^2$; against D it is $c^2$. Dividing by the corresponding lengths gives C: 0.498157, D: 0.646660, reversing the Count result. A term confined to the status document now contributes more strongly than repeated geography.

Keyword: count distinct matches

1. Text piecesQuery: Beijing processing
2. A list of numbersA=1, B=1, C=1, D=1, E=0 → A

The tie rule sees candidate order, not the reader's status intent.

Count: compare count directions

1. A list of numbersC has Beijing three times: cosine 0.670820
2. A list of numbersD has processing once: cosine 0.500000

C wins even though it is about rail travel.

TF-IDF: reweight before cosine

1. An operationIDF Beijing=1.405465; processing=2.098612
2. A list of numbersD=0.646660 > C=0.498157 → D

A relevant status document rises. Dates and synonymy remain outside this weighting rule.

Figure 9.1 — The same query changes rank when common and rare terms receive different weights. Original corpus, calculation and diagram, CC BY-SA 4.0; reproduced by KA-0.

Figure 9.1 follows that same second query through all three scoring rules. Weighting changes which source rises; it has not created a synonym dictionary or read a date field.

9.6 Compare all nine questions, including the inconvenient ones

The evaluation labels specify what counts as success before the run. A top prediction succeeds if it belongs to the query’s relevant set. When that set is empty, only abstention succeeds. Divide the number of successful cases by all nine queries, including the no-answer case. This is a diagnostic task-success fraction, not classification accuracy over tokens or language-model perplexity.

Table 9.6 — Frozen queries and their intended evidence. All current questions use the collection’s 2026-09-11 reference date.

Query	Text	Relevant cards	Intended question
1	Beijing lodging	B	Current limit
2	Beijing processing	D	Meaning of processing for an order associated with Beijing
3	750 lodging	B	Find the updated amount explicitly
4	hotel allowance	A or B	Find any version of the lodging policy, not certify today’s amount
5	600 Beijing	A	Find the archived 600-yuan rule
6	order processing	D	Explain status
7	order approval	E	Find the approval requirement; perform no action
8	Is 600 still the Beijing lodging limit?	B	Check an old premise against current policy
9	passport renewal	None	No supplied source answers it

Table 9.7 — Complete rankings from the actual deterministic run. The first card is the prediction; a dash means all scores are zero and there is no prediction. Such all-zero rows retain A–E only as a diagnostic storage order. Cosine scores are rounded to 12 decimal places for ranking ties and displayed to six; exact ties follow A–E.

Query	Keyword	BoW Count	TF-IDF	What the top choices show
1	A > B > C > D > E	B > A > C > D > E	B > A > C > D > E	Counts fix the first obsolete-source tie
2	A > B > C > D > E	C > D > A > B > E	D > C > A > B > E	IDF fixes the common-term distraction
3	B > A > C > D > E	B > A > C > D > E	B > A > C > D > E	All find the explicit updated amount
4	—	—	—	All miss the synonym query
5	A > B > C > D > E	A > C > B > D > E	A > C > B > D > E	All correctly retrieve the historical rule
6	D > E > A > B > C	D > E > A > B > C	D > E > A > B > C	All identify the status FAQ
7	E > D > A > B > C	E > D > A > B > C	E > D > A > B > C	All identify the approval FAQ
8	A > B > C > D > E	A > B > C > D > E	A > B > C > D > E	All follow the obsolete amount in the question
9	—	—	—	All correctly abstain

Keyword succeeds on 5/9, Count on 6/9 and TF-IDF on 7/9. Both authored locales reproduce these same matrices and decisions. The gain is real for this frozen diagnostic, but its carefully selected questions do not establish that each successive method is universally better.

Query 8 explains the version boundary especially clearly. Its known terms are Beijing, lodging and 600, exactly A’s count direction, so Count and TF-IDF both give A cosine 1. The word still expresses the need to check a current rule, but it has no coordinate here. A high similarity is therefore compatible with the wrong source for the task. Keep the version metadata and failure case for later work; do not silently remove A or relabel this query to improve KA-0’s score.

9.7 A retrieved document is one output type among several

The same text can enter very different tasks. Choosing an output contract prevents us from counting retrieval success as proof of an unrelated capability.

Table 9.8 — Five tasks applied to the running examples.

Task	Required output	Suitable unit of evaluation
Retrieval	Ranked document IDs, such as B then A	Whether a relevant document is retrieved at the required rank
Classification	One label for the whole question, such as travel or order-status	Correct question labels on held-out questions
Sequence labeling	One label per input position, such as amount-start and amount-inside	Position labels and complete labeled spans
Span extraction	Exact source interval for 750 yuan, with document/version	Correct boundaries, text and source; raw offsets matter
Generation	New answer text, for example a sentence explaining the current limit	Supported content, task completion and other declared answer criteria

Sequence labeling keeps a position axis that a single bag-of-words document vector has discarded. A conventional first-order hidden Markov model (HMM) describes hidden-state transitions and observations emitted at positions; choosing a state sequence can support labeling. A linear-chain conditional random field (CRF) instead models the label sequence conditional on the observed input, including relationships between neighboring labels. These are orientations on the task map, not algorithms to implement here. Neither is another name for TF-IDF or a universal replacement for the other. Rabiner, 1989, Lafferty et al., 2001

9.8 What a sparse bag still cannot preserve

BoW loses word order. Alice cancels Bob's order and Bob cancels Alice's order have the same word counts under a vocabulary preserving those names, but different actors. A vector cannot recover a distinction that its construction discarded. TF-IDF multiplies the same coordinates and therefore does not restore order.

We can add word N-gram features, such as the adjacent pair Beijing lodging. Across our five cards, the eight distinct adjacent pairs would expand the eight unigram columns to sixteen combined columns. C contains Beijing Beijing twice. These features preserve selected local order at the price of more dimensions and fewer observations per feature. They still miss unseen synonyms and relationships outside the chosen feature span.

This use of “N-gram” differs from Chapter 4’s N-gram language model. A feature says how often a particular sequence occurs in a document; a language model supplies a normalized distribution for the next token given a context, including a boundary policy. Counting adjacent pairs for retrieval does not by itself produce next-token probabilities.

The continuing project lives in code/knowledge-assistant/, with the frozen first dataset and complete outputs in data/knowledge-assistant/. 04-text-and-ka0.ipynb tests Chapter 8’s pure function, prints the count and weighted matrices, and runs every query through all three baselines. There is no gradient fitting to query labels: IDF fits corpus statistics, then evaluation reads the frozen relevant sets. The independent expectations include enumerated counts and scalar-expanded cosine formulas, so a function does not certify itself by generating its own answer key.

9.9 Chapter summary

Keyword overlap provides a transparent first source choice. Count vectors preserve repetition in a fixed document-term space; cosine controls overall magnitude. TF-IDF adds a declared, collection-dependent weighting convention and can reduce common-term distractions. KA-0 retains all three methods on one fixed collection, including synonym, obsolete-premise and no-answer cases. None of these scores supplies a source-date policy, a span extractor or permission to act.

We now have a precise representation failure: different words that serve similar roles occupy unrelated coordinates. Chapter 10 asks whether data can teach representations to share evidence, and what is still lost when a word always receives the same vector.

9.10 Exercises

1. Add a new card containing order approval approval to a separate collection variant. Give its Count row in the original eight columns. Recompute the IDF of Beijing and approval with six documents, keeping the stated smoothing formula.
2. For query order approval in the original collection, calculate Count cosine against D and E. Then calculate their TF-IDF cosine using $b$ and $c$. Explain which unchanged term is responsible for the nonzero score of the wrong FAQ.
3. A programmer computes IDF using five occurrences of Beijing instead of three containing documents. What value results under the stated formula, and why does it answer the wrong counting question?
4. An implementation ranks an all-zero query as A and counts that as an answer. Diagnose the evidence failure using queries 4 and 9; explain why the same abstention can be wrong on one and correct on the other.
5. A system retrieves B correctly and claims it has extracted the amount, generated a supported answer and approved cancellation. Which additional output contracts and checks are missing?
6. Explain why removing A from the evaluation collection would erase useful evidence. Propose a transparent future version policy while retaining the first comparison.

9.11 Reference answers

1. The new row is [0,0,0,0,0,1,0,2]. Beijing remains in three documents, so its IDF becomes $1+\ln(7/4)\approx1.559616$. Approval now occurs in two documents, giving $1+\ln(7/3)\approx1.847298$. Its two occurrences in the new card contribute only one additional containing document. All old vectors and queries must use the new fitted IDF for a fair variant comparison.
2. The Count query equals E’s row, so E has cosine 1. D shares only order: dot product 1 and denominator 2, giving 1/2. The weighted query equals E’s weighted row; E remains 1. D’s cosine is $b^2/(b^2+c^2)\approx0.394276$. The shared word order supplies that nonzero numerator; it does not mean processing and approval are interchangeable.
3. It gives $1+\ln(6/6)=1$. The five is collection frequency, counting repetitions, whereas df asks how widely the term occurs across cards. C’s three copies should count once toward df, so the correct value is $1+\ln(6/4)\approx1.405465$.
4. All zero means no represented overlap, not evidence for the first candidate. Query 4 has relevant lodging documents but no known synonym overlap, so abstention exposes a miss. Query 9 has no relevant source, so abstention is correct. The evaluation’s relevance labels, not the zero vector alone, distinguish these outcomes.
5. Retrieval provides a source identifier. Extraction must identify exact amount/unit boundaries and the source version. Generation must produce text whose claims are supported and applicable. Cancellation additionally needs an authorized action path and approval; finding a policy about approval cannot grant it.
6. A is needed for the historical query and for diagnosing obsolete-source selection. A future system may filter or rank versions according to an explicit current/historical intent and effective-date rule. Record it as a new method or dataset version, keep ka0-v1 and its errors, and compare on the same retained questions before claiming improvement.

9.12 References

• Manning, Raghavan and Schuetze, Introduction to Information Retrieval: term frequency, IDF, and queries as vectors. Support the count/collection distinction and vector-space comparison; our small corpus and all scores are original.
• Sparck Jones, 1972, A statistical interpretation of term specificity and its application in retrieval. Gives the original statistical specificity argument. The exact smoothed variant used here is separately declared, not attributed verbatim to this paper.
• scikit-learn, TF-IDF term weighting. Documents raw TF, smoothed IDF and length normalization matching this chapter’s chosen convention. The project implements the small calculation without a scikit-learn dependency.
• Rabiner, 1989, HMM tutorial. Page 261 locates the observation, state-sequence and parameter-estimation problems; used only for the task map.
• Lafferty, McCallum and Pereira, 2001, CRFs. The introduction and section 3 explain conditional sequence labeling. This is the author’s PostScript paper; no figure, benchmark or implementation is reproduced here.