Following on: retrieval isn't just "one cosine"
The last three posts built the foundation: embeddings carve text into "meaning = geometry" vectors, and a vector DB uses an ANN index like HNSW to find nearest neighbors in milliseconds. But once you actually stand up a RAG system, you find that "compute one cosine, take top-K" is only the skeleton — what really decides retrieval quality is a whole relay of algorithms on that skeleton: text is first split by a tokenizer, sparse BM25 and dense vectors each run a lane, RRF fuses them, the query may be rewritten or expanded first, and after retrieval the results get corrected, reranked, and de-duplicated.
This post takes those algorithms apart one by one, ordered by stage in the RAG pipeline, so you can see which slot each fills, what problem it solves, and how it connects to its neighbors.
The RAG retrieval pipeline: the algorithm map of this post
┌─0─┐ ┌──────1──────┐ ┌───2───┐ ┌──3──┐ ┌────4────┐ ┌───5───┐ ┌───6───┐
│tok│──▶│ sparse │ │dense/ │ │fuse │ │query │ │post- │ │rerank/│
│eni│ │ BM25 │──▶│multi- │──▶│ RRF │──▶│rewrite │──▶│retriev│──▶│dedup │──▶ LLM
│ze │ │(keyword) │ │vector │ │hybrid│ │(pre) │ │correct│ │cross- │
│BPE│ │ │ │DPR/ │ │ │ │multiQ/ │ │CRAG/ │ │encoder│
│ │ │ │ │MaxSim │ │ │ │HyDE/... │ │Self- │ │/MMR │
└───┘ └─────────────┘ └───────┘ └─────┘ └─────────┘ └───────┘ └───────┘
Note: query rewriting happens BEFORE retrieval; it's drawn mid-flow to align the diagram, but it feeds a rewritten query back into stage 1/2.
Stage 0: the tokenizer — where all retrieval begins
The first step of retrieval is splitting text into "tokens." It looks trivial but decides BM25's matching unit and the embedding model's input unit — split it wrong and everything downstream skews.
Two naive extremes: split by whole words (vocabulary explodes, and any unseen word is OOV and breaks) or split by characters (tiny vocabulary but very long sequences, thin semantics). Modern models take the middle road: subwords. The landmark is BPE (Byte Pair Encoding) [2]: start with "every character is its own token," then repeatedly scan the corpus and merge the most frequent adjacent pair into a new token, until you hit a target vocabulary size.
BPE learning (sketch):
init: l o w </w> l o w e r </w> n e w e s t </w>
most frequent pair is (e,s) → merge into "es"
then (es,t) → "est" ; (l,o) → "lo" … repeat
result: common fragments (est, lo, new) become single tokens; rare words
can still be rebuilt from characters → no more OOV
Common variants: WordPiece (used by BERT — it merges the pair that most increases the corpus likelihood, not simply the most frequent), and SentencePiece [3] (consumes raw strings without pre-segmentation, language-independent, with both BPE and Unigram-LM built in).
Why does this matter for your retrieval? A part number RC0402FR-0710KL gets split by the tokenizer into a run of subwords (say RC, 0402, FR, -, 0710, KL). BM25 matches on exactly these tokens; the embedding model also sees these tokens. The split decides whether an exact hit can happen at all — which is why domain settings often customize or at least inspect tokenizer behavior.
Stage 1: sparse retrieval — BM25
BM25 [1] (Best Matching 25) is the culmination of forty years of keyword retrieval, and still a baseline too strong to ignore. To get it, first get TF-IDF: a term's importance to a document = TF (term frequency — how often it appears here) × IDF (inverse document frequency — how rare it is across the corpus). IDF pushes ubiquitous words (the, of) toward zero weight and lifts rare, discriminative ones (10KΩ).
BM25 adds two crucial "reality corrections" on top of TF-IDF:
f(q_i, D) · (k1 + 1)
BM25(D,Q) = Σ IDF(q_i) · ─────────────────────────────────────
q_i∈Q f(q_i, D) + k1 · (1 − b + b · |D|/avgdl)
f(q_i,D) = count of term q_i in document D
|D| = document length, avgdl = average document length
k1 (typ. 1.2~2.0) = controls "TF saturation": the 10th vs 100th occurrence contribute similarly
b (typ. 0.75) = controls "length normalization": long docs shouldn't win just by being long
The intuition behind each: (1) TF saturation — more occurrences means more relevant, but not linearly; appearing 20 times shouldn't count 10× appearing twice. The fraction makes TF flatten toward a ceiling (set by k1). (2) Length normalization — a 10,000-word document is more likely to "just happen" to contain your term, so b uses length/avg-length to subtract that unfair advantage.
TF saturation curve (BM25 vs linear TF)
contrib
│ ____________ BM25 (approaches a ceiling)
│ /
│ /
│ / ····················· linear TF-IDF (rises without bound)
│/·
└──────────────────────▶ term frequency f
Because BM25 matches exact tokens, it's inherently great at "one character off means a different thing" exact hits (part numbers, process abbreviations, model codes) — precisely what dense vectors miss (recall post #2, dense vs sparse).
Stage 2: dense and multi-vector — DPR and MaxSim
The dense lane's landmark is DPR (Dense Passage Retrieval) [4]: query and passage each pass through an encoder (two-tower / bi-encoder, recall post #1), become a single vector, compared by inner product. It captures semantics (synonyms, paraphrases) but compresses detail into one vector (recall posts #1–2).
In between sits ColBERT's MaxSim [5]. It keeps a vector per token, and each query token picks its most similar token across the whole document, then sums those maxima:
MaxSim similarity:
|Q|
S(Q,D) = Σ max cos( q_i , d_j )
i=1 j∈|D|
for each query token q_i: scan all doc tokens d_j, take the best match
→ then sum the best-match scores over all q_i
intuition: each query word scores if the document contains a word it "lines up with"
This preserves more token-level correspondence than a single vector (precision near a cross-encoder) while still precomputing the doc's token vectors (cheaper than a cross-encoder) — the tier between dense and cross-encoder on the precision/cost spectrum (post #3).
Stage 3: fusion — RRF and hybrid
Now you hold two rankings: BM25's (sparse) and the vector's (dense). How to merge? You can't just add the scores — BM25 might be an unbounded real 030, cosine is −11; different scales, and a naive sum lets one drown the other.
RRF (Reciprocal Rank Fusion) [6] is clever: look only at ranks, not raw scores. In each ranking, the document at position rank gets 1/(k+rank), and you sum its scores across rankings:
1
RRF(d) = Σ ───────── k usually 60
ranking r k + rank_r(d)
e.g. doc X ranks 1 in BM25, 3 in the vector lane
RRF(X) = 1/(60+1) + 1/(60+3) = 0.0164 + 0.0159 = 0.0323
doc Y ranks 1 in the vector lane only, absent in BM25
RRF(Y) = 1/(60+1) = 0.0164
→ X is high in both lanes, so it wins the fusion
k=60 is the sweet spot Cormack et al. measured on TREC in 2009: it acts as a smoothing factor, keeps any one lane's #1 from dominating, and keeps mid/low ranks meaningful. Because it uses only ranks, RRF is inherently normalization-free and scale-agnostic — which is why nearly all hybrid search (including the BGE-M3 dense+sparse you can use, post #2) fuses with it.
Stage 4: query rewriting (act before retrieval)
Everything so far improved "how to match," but there's a more upstream problem: the way users ask often doesn't match the way documents are written (vocabulary mismatch). A user asks "why does this part keep failing," the document says "batch defect-rate anomaly analysis." This stage's algorithms all reshape the query before it's sent to retrieval.
Query Rewrite (Rewrite-Retrieve-Read) [8]: turn the classic "retrieve → read" into "rewrite → retrieve → read," letting an LLM first rewrite a colloquial, vague question into a query better suited for search (add keywords, drop noise).
Multi-Query / RAG-Fusion [11]: instead of one rewrite, have the LLM generate multiple query variants from different angles, retrieve for each, then fuse the multiple result lists with the RRF from the previous stage. Angles a single phrasing would miss get covered by others.
HyDE (Hypothetical Document Embeddings) [9]: counterintuitive but effective — first have the LLM generate a "hypothetical answer document" from thin air (it may contain errors, that's fine), then retrieve using that fake answer's embedding. The underlying assumption: in vector space, "an answer" sits closer to "the real answer document" than "a question" does — so searching answer-with-answer beats searching answer-with-question.
Step-Back [10]: first abstract the specific question into a higher-level question ("the derating curve of this 0402 resistor?" → "what is the principle of resistor derating?"), retrieve the higher-level knowledge as background, then answer the specific question. Query Decomposition breaks a multi-hop question into sub-questions retrieved separately (good when answering requires chaining several facts).
one query, several "pre-retrieval" reshapings:
┌── rewrite ─▶ a more searchable phrasing
raw query ──▶─┼── multi-Q ─▶ variants 1/2/3 ─▶ retrieve each ─▶ RRF fuse
├── HyDE ─▶ generate a fake answer ─▶ retrieve by its vector
└── step-back─▶ higher-level question ─▶ retrieve background
Stage 5: post-retrieval correction and self-reflection
What comes back isn't necessarily right. This stage makes the system check and correct retrieval instead of feeding it blindly to the LLM.
CRAG (Corrective RAG) [12]: train a lightweight retrieval evaluator that scores confidence for "query and retrieved docs," and route into three actions — Correct (trustworthy): use "decompose-then-recompose" to break docs into strips, drop the noisy strips, keep only key knowledge; Incorrect (untrustworthy): discard the batch and fall back to a web search; Ambiguous (unsure): blend both. The core value is a safety net for when retrieval fails.
Self-RAG [13]: goes further — it turns "whether to retrieve, whether the retrieved passage is relevant, whether the generation is supported by it, whether the answer is useful" into reflection tokens the model itself emits. The model self-evaluates as it generates: first decides "does this need retrieval (Retrieve)," after retrieval marks "is this relevant (ISREL)," after generation marks "is this supported by the docs (ISSUP) / useful overall (ISUSE)," using these signals to self-gate and pick the best passages. It internalizes "retrieve and critique" into the model rather than bolting on an external evaluator.
CRAG routing:
retrieved docs ─▶ [evaluator confidence]
├─ high → decompose→recompose: strip, filter noise, keep essence
├─ low → discard → trigger web search fallback
└─ mid → blend both
Stage 6: reranking and diversity
A batch of candidates comes back from coarse retrieval (say top-100); the last mile is fine ranking and de-duplication.
Cross-encoder reranking (monoBERT) [7]: concatenate the query and a candidate document into one sequence through BERT, letting both sides' tokens attend inside the network, and output a relevance probability (recall post #1, bi- vs cross-encoder). High precision but expensive — a new query means recomputing everything, no caching, so it only fine-ranks the coarse top-K, never the whole DB. Standard practice: bi-encoder/BM25 coarse top-100 → cross-encoder fine top-5.
MMR (Maximal Marginal Relevance) [14]: solves the redundancy of "the top 5 all say the same thing." It picks one at a time, each time choosing the doc that is "relevant enough to the query yet different enough from what's already picked":
MMR = argmax [ λ · sim(d, Q) − (1−λ) · max sim(d, d_j) ]
d∉S d_j∈S
large λ → favors relevance (may repeat); small λ → favors diversity (may drift)
intuition: relevance minus the max similarity to already-picked docs (a redundancy penalty)
If the context handed to the LLM is both relevant and non-repetitive, answer coverage improves — especially useful when "one part has many near-identical work orders."
Landing it in your pgvector RAG: one full retrieval chain
Stringing it all together, mapped to your MES / part-number scenario:
text ─[tokenizer]─┬─[BM25 sparse]──────┐
└─[embedding dense]──┤
(query side may first rewrite / multi-query / HyDE)
▼
[RRF fusion] ─▶ top-100 candidates
▼
[cross-encoder rerank] ─▶ top-k
▼
[MMR dedup] (optional)
▼
[CRAG gatekeeping] (optional: low conf → web/discard)
▼
LLM generation
Priority in practice: first nail the BM25 + vector + RRF hybrid (best bang for buck — exact part-number hits plus semantic generalization at once); add a cross-encoder reranker if candidate precision falls short; bring in query rewrite / multi-query when phrasings are erratic or cross-lingual; only for high-stakes QA where retrieval failure is costly is a correction layer like CRAG / Self-RAG worth it. On pgvector, do sparse via Postgres full-text search or a BM25 extension, dense via <=>, and fuse the two rankings with RRF in the app layer.
Closing the loop with one causal chain
Text is first tokenized into subwords (setting the matching unit) → BM25 does exact-token matching with TF saturation + length normalization, DPR/MaxSim do semantic matching with vectors → the two lanes have different scales, so fuse them with rank-only RRF (normalization-free) → but user phrasing often mismatches the docs, so pre-retrieval rewrite/multi-query/HyDE/step-back reshape the query → what comes back isn't always right, so CRAG/Self-RAG evaluate, correct, even switch to web → finally cross-encoder fine-ranks and MMR de-duplicates, handing the LLM context that's both accurate and non-repetitive.
In one line: RAG retrieval isn't a single model but a pipeline you can swap and tune stage by stage; each slot patches the weakness of the one before it.
Extension hooks
- Tuning BM25: how to set k1, b for your short part-number descriptions vs long spec sheets, and how they interact with tokenization.
- Learned sparse (SPLADE) replacing BM25: letting a model learn "which term matters," vs BM25 (continues post #2).
- Choosing a reranker: cross-encoder vs LLM-as-reranker (listwise, RankGPT/RankZephyr), the latency/quality trade-off.
- Deployment cost of CRAG/Self-RAG: whether to run an extra model for that evaluator, and at what scale it pays off.
- Chunking strategies: sentence-window, parent-document, RAPTOR tree summaries — the split often affects recall more than swapping models.
Tell me which to go deep on.
References
[1] Robertson, S., Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in IR. — https://doi.org/10.1561/1500000019
[2] Sennrich, R., Haddow, B., Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units (BPE). — https://arxiv.org/abs/1508.07909
[3] Kudo, T., Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer. — https://arxiv.org/abs/1808.06226
[4] Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering (DPR). — https://arxiv.org/abs/2004.04906
[5] Khattab, O., Zaharia, M. (2020). ColBERT: Contextualized Late Interaction over BERT (MaxSim). — https://arxiv.org/abs/2004.12832
[6] Cormack, G., Clarke, C., Buettcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR. — https://doi.org/10.1145/1571941.1572114
[7] Nogueira, R., Cho, K. (2019). Passage Re-ranking with BERT (monoBERT). — https://arxiv.org/abs/1901.04085
[8] Ma, X. et al. (2023). Query Rewriting for Retrieval-Augmented Large Language Models (Rewrite-Retrieve-Read). — https://arxiv.org/abs/2305.14283
[9] Gao, L. et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE). — https://arxiv.org/abs/2212.10496
[10] Zheng, H. et al. (2023). Take a Step Back: Evoking Reasoning via Abstraction in LLMs (Step-Back). — https://arxiv.org/abs/2310.06117
[11] Rackauckas, Z. (2024). RAG-Fusion: a New Take on Retrieval-Augmented Generation. — https://arxiv.org/abs/2402.03367
[12] Yan, S.-Q. et al. (2024). Corrective Retrieval Augmented Generation (CRAG). — https://arxiv.org/abs/2401.15884
[13] Asai, A. et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. — https://arxiv.org/abs/2310.11511
[14] Carbonell, J., Goldstein, J. (1998). The Use of MMR, Diversity-Based Reranking for Reordering Documents. SIGIR. — https://doi.org/10.1145/290941.291025
[15] Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG). — https://arxiv.org/abs/2005.11401