Chapter 44 Modern Retrieval: BM25, Dense, Hybrid, and Rerankers

44.1 Which allowed document should reach the answerer?

MCP can deliver a search request, but it supplies no ranking rule. Our collection still contains both 600 and 750, a rail FAQ and two order FAQs. Returning everything transfers the selection problem to a generator with a finite context budget. Returning the wrong first item can be worse than returning nothing.

We resume the exact KA-0 collection and nine queries from Chapters 9 and 10, keeping the relevance labels and three lexical baselines. “Relevant” means useful for the specified information need, not merely linguistically similar. All following local results use five fictional index cards, not an industrial benchmark.

44.2 Repeated words should help, but not without limit

Example 44.1 — Score the current-lodging query. Query terms are beijing lodging. The shared tokenizer maps the bilingual cards to the same canonical terms. We score cards, not full policy bodies; the exact indexing boundary remains KA-0’s.

Table 44.1 — The entire indexed collection.

Label	Source	Canonical card	Length in terms
A	travel-v1	beijing lodging 600	3
B	travel-v2	beijing lodging lodging 750	4
C	rail-faq-v1	beijing beijing beijing train	4
D	status-faq-v1	order processing	2
E	approval-faq-v1	order approval	2

The five lengths sum to 15, so average length is $15/5=3$. Beijing appears in three documents; lodging appears in two. Raw word counts reward repetitions linearly. BM25 instead combines a term’s rarity with a saturating frequency contribution and a soft document-length adjustment.

We choose the positive IDF variant $\ln(1+(N-df+0.5)/(df+0.5))$. With $N=5$, Beijing’s weight is $\ln(12/7)=0.538997$, and lodging’s is $\ln(12/5)=0.875469$. Here $df$ counts documents containing the term, not total occurrences. The added 1 inside the logarithm keeps this chosen variant positive; it is not the only historical BM25 convention.

Use teaching settings $k_1=1.2$ and $b=0.75$. For B, whose length is 4, the length factor is $1-0.75+0.75(4/3)=1.25$. Beijing occurs once, so its multiplier is $1(1.2+1)/(1+1.2\times1.25)=2.2/2.5=0.88$. Lodging occurs twice, giving $2(2.2)/(2+1.5)=4.4/3.5$. Add the two weighted contributions:

$$0.5389965\times0.88+0.8754687\times\frac{4.4}{3.5}\approx1.574906.$$

The general score simply repeats that calculation for every distinct query term:

$$\operatorname{BM25}(q,d)=\sum_{t\in q}\ln\!\left(1+\frac{N-df_t+0.5}{df_t+0.5}\right) \frac{f_{t,d}(k_1+1)}{f_{t,d}+k_1(1-b+bL_d/\overline L)}.$$

$q$ is the set of query terms, $d$ one document, $f_{t,d}$ the number of occurrences of term $t$ in it, $L_d$ its length and $\overline L$ the collection mean. Lengths use the same term units, so their ratio has no units. The score is a ranking value, not a probability. A missing term has frequency zero and contributes zero. Repeated query terms count once in this implementation.

Table 44.2 — Reproduce every candidate score.

Card	Beijing contribution	Lodging contribution	Total
A	0.538997	0.875469	1.414465
B	0.474317	1.100589	1.574906
C	0.790528	0	0.790528
D	0	0	0
E	0	0	0

Thus the full order is B, A, C, D, E; only positive-score candidates are returned. For an average-length document, frequency 1, 2 and 3 yields multipliers 1, 1.375 and about 1.571429. Each repetition adds less. Increasing length while holding frequency fixed reduces the contribution when $b>0$; setting $b=0$ removes that adjustment. Neither change teaches a synonym or knows which policy is current.

The two settings control these effects. With positive $k_1$ and a positive length factor, the denominator is greater than the frequency alone. The multiplier therefore stays below $k_1+1$, here 2.2, rather than growing without limit. $k_1$ sets this saturation scale. The usual range $0\le b\le1$ interpolates between no length adjustment at 0 and the full length ratio at 1; 0.75 lies between them. These are ranking choices to evaluate, not facts about the policy.

44.3 A vector representation changes matching, not authority

A query saying “hotel allowance” shares no indexed canonical term with the lodging cards. BM25 returns nothing. An embedding model maps text to vectors; nearby vectors may connect such paraphrases. An index stores searchable representations. A retriever uses a query and index to return ranked candidates. These are different components: replacing the model may require rebuilding the index, while replacing exhaustive search with an approximate index can change recall without changing embeddings.

The local comparison reuses two Chapter 10 representations. One is an explicitly authored three-axis map: lodging, Beijing, hotel and allowance point along the first axis; order terms along the second; train along the third; amount terms are zero. Mean pooling gives A=$[2/3,0,0]$, B=$[3/4,0,0]$, and query “hotel allowance”=$[1,0,0]$. Cosine is 1 for both A and B. It finds the topic but cannot distinguish amounts or dates. D and E likewise collapse to the same direction.

The other representation is the retained four-dimensional NNLM embedding learned in Chapter 10, averaged and compared by cosine. Reusing saved learned vectors is a real numerical retrieval calculation, not new training or a pretrained DPR reproduction. Its unknown synonym query is a zero vector and returns nothing. A score from either representation is not evidence of authorization or factual support.

44.4 Fusion changes the candidate order; reranking revisits it

Example 44.2 — Combine ranks, not incompatible score units. BM25 returns B,A,C on the first query; the authored dense map returns A,B,C. Hybrid retrieval combines lexical and dense evidence. Our reciprocal rank fusion (RRF) gives each returned document $1/(60+r)$ from each list, where $r$ starts at 1. A missing document contributes zero.

A and B each receive $1/61+1/62\approx0.032522$; C gets $2/63\approx0.031746$. The tie breaks by the original A–E order, so hybrid returns A,B,C. Fusion has made this current-policy query worse. The offset 60 is a declared choice, not a learned relevance probability or a guarantee of improvement.

A reranker re-examines a small candidate set against the question. A neural cross-encoder can jointly read question and passage, unlike independently computed embeddings; doing that for many pairs costs additional model work. It cannot recover an item absent from its candidate set.

Our executable reranker is a deliberately narrow rule, not a neural model. It takes the top three hybrid candidates and stably moves topic/date-compatible ones first. “processing” selects order scope; “approval” selects approval scope; lodging and amount terms select travel. “still,” “current,” or 750 implies the fixture’s current date; 600 without a current cue implies its historical date. The bilingual cue lists are saved with the rules. No query ID or relevance label is read by the ranking function. Ambiguous wording is a limitation: bare “Beijing lodging” supplies no date cue, so the rule leaves the harmful A/B tie unresolved.

Same query: Beijing lodging

1. An operationBM25: B → A → C
2. Text piecesDense: A → B → C
3. An operationRRF: A = B; tie → A

Higher coverage can coexist with a worse first answer source.

Shortlist boundary

1. Outside the modelCandidates C, D, E
2. An operationRerank only these items
3. Outside the modelRequired B still absent

Repair retrieval or filtering before tuning the last scoring stage.

Figure 44.1 — Fusion can move an obsolete source upward. Original teaching diagram · CC BY-SA 4.0

Figure 44.1 locates representation, candidate retrieval, fusion and reranking. In the frozen experiment, every method sees the same questions and cards. Rule design has used this tiny development collection; its scores are diagnostics, not held-out generalization.

44.5 Read complete rankings before reading averages

Table 44.3 — All returned candidates, in order, for both languages. A dash means no positive candidate; the exact canonical bilingual mapping makes these runs equal. Stored full score orders also include nonpositive candidates, which are never silently treated as retrieved evidence.

Query	Relevant cards	BM25	Authored dense	Learned NNLM	Hybrid	Rule rerank
Beijing lodging	B	B A C	A B C	B A C	A B C	A B C
Beijing processing	D	D C A B	A B D E C	D E C B	A D B C E	D A B
750 lodging	B	B A	A B C	B A D E	A B C	B A C
hotel allowance	A, B	—	A B C	—	A B C	A B C
600 Beijing	A	A C B	A B C	A B	A B C	A B C
order processing	D	D E	D E	D E C B	D E	D E
order approval	E	E D	D E	E D C	D E	E D
Is 600 still the Beijing lodging limit?	B	A B C	A B C	A B	A B C	B A C
passport renewal	None	—	—	—	—	—

Top-one success, including correct no-result behavior, is 7/9 for BM25, 4/9 for authored dense, 7/9 for learned NNLM, 4/9 for hybrid and 8/9 for rule reranking. The retained Keyword, BoW Count and TF-IDF baselines are respectively 5/9, 6/9 and 7/9. None has been removed because a later method scores poorly. “Hotel allowance” accepts either travel version under the original labels; that must not be silently changed into a current-policy query.

44.6 Count coverage, first useful rank and graded ordering separately

Example 44.3 — A new grading condition. To expose the metrics, use an authored query asking for both versions, preferring the current one. Assign A grade 1, B grade 2, and C,D,E grade 0. Suppose the returned ranking is C,A,B. These grades are a separate worked example, not altered KA-0 labels.

At depth $k=2$, only A is retrieved among the two relevant cards, so Recall@2 is $1/2$. Its denominator is all relevant cards, not the two returned positions. The first relevant card is at rank 2, so reciprocal rank is $1/2$. MRR averages such reciprocal ranks across queries; it ignores how many additional relevant cards come later.

For discounted cumulative gain (DCG), choose gain $2^g-1$ for grade $g$ and discount $\log_2(r+1)$ at rank $r$. Grade 2 gives gain 3; grade 1 gives 1; grade 0 gives 0. At two positions, C contributes zero and A contributes $1/\log_2 3\approx0.630930$. The best possible order is B,A, with gain $3+1/\log_2 3\approx3.630930$. Divide actual by ideal:

$$\operatorname{nDCG@2}=\frac{0.630930}{3.630930}\approx0.173765.$$

The denominator asks how good the best two-position ranking could be under the same labels. It is not a document count. At depth 3, B adds $3/\log_2 4=1.5$, yielding nDCG about 0.586883; Recall becomes 1, but the late best document still costs discounted gain.

Table 44.4 — Macro metrics on the eight nonempty-gold KA-0 queries. Binary grade 1 is used for each relevant document here; figures are actual calculations rounded to six decimals. Reciprocal rank uses the full returned list.

Method	Recall@2	MRR	nDCG@2
BM25	0.875000	0.812500	0.828866
Authored dense	0.875000	0.666667	0.690465
Learned NNLM	0.875000	0.812500	0.828866
Hybrid	1.000000	0.687500	0.769331
Rule rerank	1.000000	0.937500	0.953866

An empty relevant set would divide Recall and normalized gain by zero. We mark these metrics not applicable, exclude that query from their macro means and separately report correct abstention: 1/1 for every method. A nonempty relevant set with no returned candidate scores zero. Returning duplicate documents cannot inflate recall; the evaluator rejects duplicate IDs. These conventions must accompany the averages.

44.7 KA-4: a searchable collection is still not an answer

retrieval.py reproduces the rankings from frozen cards, tokenization and representation artifacts. It retains full scores, positive returned lists, ties, labels and denominators in data/part-vii/retrieval-run.json. The second Notebook reproduces scores and metrics on CPU; no external embedding endpoint or vector database is required. Exact scan is appropriate for five vectors, while a large index has additional memory and approximation trade-offs.

A vector database stores and searches vectors. It does not ensure the right document version, sufficient context, faithful generation or supporting citations. We have evaluated candidate retrieval. Chapter 45 follows selected passages through the rest of an answer pipeline.

44.8 Summary

BM25 changes lexical weighting; dense representations change matching; hybrid fusion combines candidate rankings; reranking re-evaluates a shortlist. Each has a different failure boundary. Recall, first useful position and graded ordering answer different evaluation questions. Retaining the lexical baselines and complete rankings makes those differences inspectable.

44.9 Exercises

1. For an average-length card, calculate the BM25 frequency multiplier at frequency 4 with $k_1=1.2$. Compare it with frequency 2. Does doubling frequency double the multiplier?
2. Fuse lists B,A and A,C with offset 60. Show each score and the final order under A–E ties.
3. Using Example 44.3’s grades, evaluate ranking B,C,A at depth 2: Recall, reciprocal rank and nDCG. Explain why perfect reciprocal rank does not mean complete evidence.
4. A reranker only receives C,D,E while B is the sole relevant document. Can a more expensive reranker fix recall? What component must change?

44.10 Reference answers

1. The multiplier is $4(2.2)/(4+1.2)=8.8/5.2\approx1.692308$. At frequency 2 it is 1.375. Doubling adds about 0.317308, not another 1.375, because of saturation.
2. A gets $1/62+1/61\approx0.032522$, B gets $1/61\approx0.016393$, C gets $1/62\approx0.016129$. The order is A,B,C. No relevance judgment follows from these rank sums alone.
3. Recall is $1/2$, reciprocal rank is 1, DCG is 3 and ideal DCG is 3.630930, so nDCG is about 0.826235. The first result is useful, but A is outside the cutoff; evidence coverage remains incomplete.
4. No. Reordering cannot introduce B. Change initial retrieval, candidate depth or an erroneous filter, then assess the extra cost and recall. Do not blame the last scoring stage for a missing candidate.

44.11 References

• Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond, 2009, §§3.4–3.5, and Trotman et al., Improvements to BM25 and Language Models Examined, 2014, §2. Explain saturation, length normalization and formula variants.
• Karpukhin et al., Dense Passage Retrieval, 2020, §3. Explains learned query/passage encoders; the book’s small mean-vector baselines do not reproduce DPR.
• Cormack et al., Reciprocal Rank Fusion, 2009. Supplies the rank-fusion mechanism and offset provenance.
• Nogueira and Cho, Passage Re-ranking with BERT, §2. Locates joint question/passage scoring after candidate retrieval.
• Manning et al., Introduction to Information Retrieval, §8.4. Explains ranked and graded evaluation; our exact denominator and no-result conventions are stated above.