Start with the problem: retrieval brings it back, so why is it still ranked wrong
Push the problem to the bottom. Your RAG pipeline does vector retrieval with a bi-encoder (two-tower embedding model): the query passes through the model once to become a vector, the millions of document vectors in your warehouse were computed offline and stored in pgvector, and online you only run one nearest-neighbor search. Fast — genuinely fast. But you've surely hit this: query "RC0402 overheating derating curve," and the correct answer is in the returned top-10 — sitting at rank 7, with several "merely also mentions resistors" half-relevant documents stuffed above it.
Why? It's not that your embedding model is weak — the bi-encoder has a precision ceiling built into its mechanism, and that ceiling comes from the very design it relies on for speed:
- The whole document is squeezed into one fixed-length vector. 768 dimensions must hold all the semantics of a passage, so detail is necessarily lossy — fine-grained signals like "which word modifies which" or "the difference between 0402 and 0603" get averaged away the moment mean pooling happens.
- Query and document meet for the first time only at scoring. Each side is encoded independently; when encoding the document, the model has no idea what the query will ask, so it can only produce a "generic summary vector." Scoring is one cosine/dot product between two vectors — no token-level interaction whatsoever.
So the bi-encoder's strength is recall — at the price of precomputability and indexability, it quickly fishes "roughly relevant" candidates out of millions; its weakness is precision — it can't tell "actually answers the question" from "merely on the same topic."
Reranking is the second stage born to close exactly that gap: let the cheap retriever pull 100 from 1,000,000 (owning recall), then let an expensive-but-accurate model carefully re-order just those 100 (owning precision). This "coarse filter → fine re-rank" two-stage architecture is the skeleton of this whole article:
Stage 1 (recall) Stage 2 (precision)
1,000,000 docs ──────────────────▶ 100 docs ──────────────────▶ 5 docs
bi-encoder/BM25 reranker
fast, precomputable, coarse slow, non-precomputable, accurate
goal: answer must be inside goal: answer must be on top
The next questions follow naturally: what makes the reranker more accurate? What exactly is its score, and how is it computed? — that's the core of this article.
The protagonist: the cross-encoder — how the score actually gets computed
Today's most mainstream reranker is the cross-encoder, first implemented with BERT by Nogueira & Cho (2019) (monoBERT)[1]. The best way to understand it is to dissect it side-by-side with the bi-encoder.
How the input goes in. In a bi-encoder, query and document each walk their own tower; a cross-encoder instead concatenates both into one sequence and feeds it through the Transformer together:
bi-encoder (two towers):
query ──▶ [Encoder A] ──▶ vector q ─┐
├─▶ cosine(q, d) = score
doc ──▶ [Encoder B] ──▶ vector d ─┘
(neither side knows the other exists; they touch once at the end)
cross-encoder (one tower):
[CLS] query tokens [SEP] document tokens [SEP]
└────────── the whole thing fed into one Transformer ──────────┘
What happens inside. This step is the entire secret of the cross-encoder's accuracy. Recall the self-attention mechanism: every token in the sequence attends to every other token. Because query and document now share one sequence, from layer 1 onward every query word can directly attend to every document word — the query token "overheating" can align directly with "thermal derating" in the document, "0402" can line up with the document's "0402," and it can also notice the document actually says "0603" and lower the match. This is early interaction: word-to-word matching signals accumulate and get refined layer by layer inside the network, instead of being squeezed into two isolated vectors that only meet at the end like the bi-encoder.
Where the score comes out. After all the layers, take the final hidden vector of the [CLS] token at the start of the sequence (it has attended across the whole query+document pair all the way up, making it an aggregate representation of the pair's relationship), then project it to a scalar through a linear layer (one weight vector w):
score(q, d) = w · h_[CLS] + b ← one real number, that's it
h_[CLS]: final hidden vector at the [CLS] position (e.g. 768-dim)
w, b: trained linear-layer weight and bias
That number is the relevance score of reranking. Note its properties — this is where practice most often goes wrong:
- It is a logit (unnormalized raw score), unbounded (could be -8.3 or 5.7) — not a probability, not 0–1. To get 0–1, pass it through a sigmoid:
p = 1/(1+e^(-score)). - It is only comparable among candidates of the same query. Under one query, doc A scoring 3.2 vs doc B scoring 1.1 means A is more relevant; comparing scores across queries is meaningless, because the model was never trained to be calibrated across queries.
- The "ranking" act is exactly this: compute this score for each of the top-K candidates, then sort descending by score — reranking's "rank" is that plain; all the craft is in how the score is computed.
What carves the score (the objective layer). Mechanism alone isn't enough — the linear layer and all Transformer weights are carved by a loss. The standard recipe (monoBERT[1] and bge-reranker[9] alike): take a human-annotated retrieval dataset (like MS MARCO — about a million real Bing queries with labeled relevant passages[7]), form (query, positive doc, negative doc) triples, and train with binary cross-entropy to push positives' scores up and negatives' down. The critical detail is how negatives are picked: random negatives are too easy (the model learns nothing); you need hard negatives — documents "retrieved by BM25 or a bi-encoder, looking very similar but actually wrong." This maps exactly onto your pain point: the reranker can distinguish "actually answers it" from "merely similar topic" precisely because training forced it to discriminate on such hard pairs. That's the three-layer framework again — the accuracy isn't given by the attention mechanism itself; the hard-negative loss carved attention's weights into a shape that does fine-grained matching.
Then why is it slow? Also derived from the mechanism. The score depends on the whole (query, doc) pair — change the query and everything computed before is void. Hence:
- Nothing can be precomputed. A bi-encoder's document vectors can be computed offline and indexed; a cross-encoder has no "document vector" to store.
- Every candidate costs one full forward pass. Reranking 100 docs = 100 Transformer forward passes, each over the concatenated query+document length, with O(L²) attention cost.
This accurate-but-slow combination locks in the cross-encoder's role: second-stage re-ranker over top-K candidates only; never a full-corpus scanner. Stage 1 (BM25/bi-encoder) and stage 2 (cross-encoder) aren't competitors — they're mechanically complementary divisions of labor.
The middle road: ColBERT's late interaction and MaxSim
Bi-encoder too coarse, cross-encoder too expensive — ColBERT by Khattab & Zaharia (2020)[3] offers a middle road, with the principle encoded in the name: Contextualized Late Interaction over BERT.
Its approach: documents are still encoded offline (keeping the precompute advantage), but not squeezed into one vector — every token's vector is kept. Online, the query is also encoded into a string of token vectors, and interaction happens through a very cheap operator — MaxSim:
score(q, d) = Σ_i max_j ( E_qi · E_dj )
term by term:
E_qi : vector of the query's i-th token
E_dj : vector of the document's j-th token
max_j(·) : for query token i, scan all document tokens,
find the most similar one, take that similarity
("overheating" finds "thermal derating" in the doc — takes that vote)
Σ_i : sum each query token's "best-match score" = total score
Intuition: each query word goes and finds its own best counterpart in the document; the total score is the sum of every word's best match. Token-level matching is preserved (finer than a single vector), while interaction is deferred to a final dot product (cheaper than a cross-encoder, and document vectors are precomputable and indexable). The cost is storage blowup — each document stores hundreds of vectors instead of one, inflating the index by orders of magnitude. ColBERT can serve as a (faster-than-cross-encoder) reranker, or as a stage-1 retriever with a vector index.
Generative scoring: monoT5 and LLM rerankers
The score doesn't have to come from a classification head — another route uses a generative model's token probability as the score.
monoT5 (Nogueira et al., 2020)[4]: recast reranking as text generation. Feed T5 this template:
input: "Query: q Document: d Relevant:"
output: the model generates one word — "true" or "false"
score = softmax over the logits of {true, false} at the output position,
take P("true") as the relevance score (naturally 0–1, sortable across candidates)
It's still "query+document into the model together" early interaction (so it belongs to the cross-encoder family) — only the scoring head changed from a linear layer to "the probability of the token true." The benefit: it inherits seq2seq pretraining, and in data-poor regimes it clearly beats a BERT classification head[4].
LLM rerankers (the RankGPT line) (Sun et al., 2023)[5]: skip training a dedicated model, just prompt an LLM to rank. First fix a taxonomy we'll reuse — three granularities of scoring:
- pointwise: look at one (q, d) pair at a time, emit one score. monoBERT, monoT5, bge-reranker.
- pairwise: look at (q, d_A, d_B), answer "which of A/B is more relevant." Accurate but O(N²) comparisons.
- listwise: look at (q, [d_1...d_20]) at once, output the whole ranked order directly.
RankGPT goes listwise: stuff 20 candidates plus the query into a prompt and ask the LLM to generate the ordering directly (e.g. [3] > [1] > [7] > ... — permutation generation). When candidates exceed one prompt, use a sliding window: rank 20 at a time, slide by 10, and the overlap lets good documents "bubble up" to the front. Experiments show GPT-4 doing this can beat supervised dedicated rerankers, and distilling GPT's orderings into a 440M model also performs strongly[5]. The cost is obvious: every rerank is one (or several) LLM calls — an order of magnitude more latency and cost, fit for offline evaluation or quality-critical queries.
Tuning the knobs, one by one
The part you specifically asked for. Every knob along the rerank segment of the pipeline, what it means, how to tune it, and the mechanistic reason behind it:
Knob ① Stage-1 recall count top_k (candidates sent to the reranker; typically 50–200). This knob sets the recall ceiling: the reranker can only reorder what you give it — if the correct answer wasn't fished into top_k by stage 1, no reranker can save it. Tune by the Recall@k curve: sweep k from 20 to 200, measure "fraction of queries whose answer lands in top-k," and find the knee where the curve flattens. Too small → ceiling kills you; too large → you pay the reranker's linear cost for nothing (each extra doc is one more forward pass). Empirical starting point: 100.
Knob ② Final keep count top_n (docs sent to the LLM after rerank; typically 3–10). This governs the LLM's context quality. More is not better — Liu et al.'s (2023) "Lost in the Middle" experiments show LLMs use information in the middle of the context markedly worse; stuff 20 docs in and a mid-positioned correct answer tends to get ignored[6]. Tune from 5, matched to your chunk size and context budget; and use the rerank order — put the highest-scoring docs at the very front (or front+end) of the prompt to dodge the middle blackhole.
Knob ③ max_length and long-document strategy. Cross-encoders have an input cap (commonly 512 tokens); if query+document exceeds it, it gets truncated — the cut-off part effectively doesn't exist, and the score reflects only the first half. Three counters: (a) align chunk size to the reranker's window at indexing time; (b) split long docs into passages, score each against the query, aggregate by max-passage (highest passage score = document score); (c) use a long-input reranker (the bge-reranker-v2-m3 line supports long text[9]). Truncation is the most common invisible bug in rerank practice: when scores look weird, first check whether the answering passage got cut off.
Knob ④ Batch size and latency budget. Total latency for reranking 100 docs ≈ (100/batch_size) × per-batch inference time. On GPU, a large batch squeezes 100 docs into tens of milliseconds; on CPU, honestly trade off "candidate count × model size." For latency-sensitive online serving, a common combo is "small model (e.g. bge-reranker-base) + top_k=50"; save the big model and large candidate sets for offline batches (e.g. nightly FAQ re-ordering).
Knob ⑤ Score threshold (refusal and filtering). Since the score is an unbounded logit, first sigmoid it to 0–1, then set a threshold: drop all candidates below it, and if none survive → trigger a refusal ("the knowledge base has no answer for this") instead of force-feeding bad context to the LLM and inviting hallucination. How to set it: take a batch of labeled queries, plot the score distributions, pick a value at the boundary between the positive and negative distributions, and adjust by business tolerance (prefer over-refusing or over-answering). This knob is one of the cheapest hallucination reducers in RAG.
Knob ⑥ Model selection. A fast-to-accurate spectrum; pick by latency budget:
bge-reranker-base ──▶ bge-reranker-large ──▶ bge-reranker-v2-m3 ──▶ LLM-based
fast, sufficient more accurate, slower multilingual, long-text most accurate, priciest
online high QPS online moderate QPS Chinese/multilingual offline/high-value queries
(The bge-reranker line is the mainstream open-source multilingual choice, and BAAI states the positioning plainly: "use/fine-tune them to re-rank top-k documents returned by embedding models"[8][9]. On the commercial-API side there are options like Cohere Rerank.)
Knob ⑦ Fine-tuning (when the general model fails in your domain). In verticals like part numbers and process terminology, general rerankers often can't tell RC0402 from RC0603. Recipe: collect (query, clicked/adopted doc) positives from your retrieval logs, mine hard negatives with your own stage-1 retriever (highly ranked but not adopted), positive:negative around 1:4–1:8, fine-tune with cross-entropy. Tie-back to the PEFT article: the reranker body is a BERT-class model — LoRA-fine-tuning it is cheap.
Designing your rerank: a decision checklist and the full flow
Assemble the knobs into a design process by answering four questions in order:
Q1 Do you need rerank at all? Measure before adopting. Take 50–100 queries with ground truth, measure current MRR@10 (mean reciprocal rank of the answer — "how high does the answer rank") and NDCG@10 (rank quality with graded relevance). If answers are usually inside top-50 but not top-5 — that's exactly the shape rerank fixes; if answers are often not even in top-50 — the problem is stage-1 recall (fix chunking, embeddings, hybrid retrieval); rerank can't save that.
Q2 Where does rerank sit in the pipeline? Iron rule: after all retrieval sources are fused, before the LLM. With BM25+vector hybrid retrieval, first fuse the branches with RRF (reciprocal rank fusion — rank-only fusion, dissected in this site's RAG retrieval article) into one candidate list, then rerank. The reason is mechanistic too: RRF only sees ranks and throws away semantic detail — exactly what the cross-encoder adds back with fine-grained matching; the reverse order (rerank each branch, then fuse) multiplies your rerank cost.
Q3 What does the full RAG + rerank flow look like?
user query: "RC0402 overheating derating curve?"
│
├──▶ BM25 (jieba segmentation) ─▶ top-100 (strong lexical match: exact part number)
│ │
└──▶ bi-encoder vector search ──▶ top-100 (strong semantic match: "overheat"≈"thermal derating")
│
RRF fusion, dedup ─▶ ~150 candidates
│
┌──────────────────────────▼──────────────────────────┐
│ RERANK: cross-encoder │
│ for each doc: score = w·h_[CLS]([q ; doc]) │
│ 150 forwards → 150 logits → sort descending │
│ sigmoid, threshold=0.35 → 8 docs survive │
└──────────────────────────┬──────────────────────────┘
│ take top_n=5
prompt assembly (highest first, dodge lost-in-middle)
│
LLM generation (with citations)
Q4 How to validate after launch? Controlled comparison: same test queries, rerank off vs on, compare MRR@10, NDCG@10, and end-to-end answer accuracy; watch P95 latency alongside. The payoff formula is simple: if stage-1 recall is good (answer inside top_k) but ordering is bad, the payoff is large; if recall itself is broken, don't pay for rerank yet.
Cross-comparison table
Approach Interaction timing Where the score comes from Precompute Latency Role
──────────────────────────────────────────────────────────────────────────────────────────
BM25 none (term stats) TF-IDF weighted sum yes(inverted) v.low stage 1 (lexical recall)
bi-encoder none (late touch) cosine(q vec, d vec) yes(vectors) v.low stage 1 (semantic recall)
ColBERT late interaction MaxSim sum of best matches yes(multi-vec, fat) low stage 1 or light rerank
cross-encoder early interaction logit of w·h_[CLS] no med-high stage 2 rerank (mainstream)
monoT5 early interaction P("true") generation prob no med-high stage 2 rerank
LLM reranker early (in prompt) listwise permutation gen no v.high offline/high-value rerank
One line to grab the axis: the later query and document interact, the more precomputable, faster, and coarser; the earlier they interact, the more accurate, more expensive, and more strictly second-stage. Designing rerank is choosing a point on this spectrum for your latency budget.
Closing: one logic chain tying the whole article together
The bi-encoder, to stay precomputable, squeezes documents into single vectors and lets query and document meet only at the end → fine-grained matching signals are lost at pooling, so it can't tell "actually answers" from "similar topic" → the fix is two stages: a cheap retriever fishes top-K, an expensive accurate reranker re-orders → the cross-encoder concatenates query+document into one sequence, self-attention aligns every word pair from layer 1 (early interaction), and [CLS]'s final vector through a linear layer emits one logit — the relevance score → sort descending and the rerank is done; and the score "understands" fine distinctions because MS MARCO + hard negatives' cross-entropy carved the weights that way → the same early interaction can also score via generation probability (monoT5's P("true")) or let an LLM emit a listwise permutation (RankGPT) → ColBERT stands in the middle with MaxSim: token-level matching kept, interaction deferred, vectors precomputable → what actually decides success in engineering are the knobs: top_k sets the recall ceiling, top_n dodges lost-in-the-middle, max_length prevents truncation bugs, threshold enables refusal, model choice and fine-tuning align the domain → design order: measure Recall@k and MRR to confirm the disease is ordering, place rerank after RRF fusion and before the LLM, and validate with NDCG/MRR + latency comparisons.
One line for the whole article: rerank is the second stage that trades early interaction for precision — the cross-encoder lets query and document meet word-by-word inside attention and emits a hard-negative-carved logit; sort it, and that's the rerank. All the engineering is turning those seven knobs among the recall ceiling, the latency budget, and context quality.
Extension hooks
To go deeper, pick from: how hard-negative mining strategies (BM25 negatives vs in-batch negatives vs distillation) affect reranker accuracy; how ColBERT's vector compression and PLAID indexing slim down the fat index; listwise losses (the RankNet/LambdaRank family) vs pointwise cross-entropy; and using rerank scores as RAG confidence signals to drive CRAG-style self-correcting retrieval. Any one could be its own article.
References
[1] Nogueira, R., Cho, K. (2019). Passage Re-ranking with BERT (monoBERT, the start of cross-encoder reranking). — https://arxiv.org/abs/1901.04085
[2] Reimers, N., Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (the bi-encoder contrast). — https://arxiv.org/abs/1908.10084
[3] Khattab, O., Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. — https://arxiv.org/abs/2004.12832
[4] Nogueira, R., Jiang, Z., Lin, J. (2020). Document Ranking with a Pretrained Sequence-to-Sequence Model (monoT5). — https://arxiv.org/abs/2003.06713
[5] Sun, W. et al. (2023). Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (RankGPT). — https://arxiv.org/abs/2304.09542
[6] Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. — https://arxiv.org/abs/2307.03172
[7] Bajaj, P. et al. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. — https://arxiv.org/abs/1611.09268
[8] Xiao, S. et al. (2023). C-Pack: Packed Resources For General Chinese Embeddings (the BGE line). — https://arxiv.org/abs/2309.07597
[9] BAAI (2023–). FlagEmbedding: the bge-reranker line (base/large/v2-m3/v2-gemma etc.). — https://github.com/FlagOpen/FlagEmbedding