Following on from post #4: this time we walk the tokenizer and ranking flow end-to-end
Post #4 was the algorithm map; this one is hands-on. I'll answer three very concrete questions: (1) why the moment you have professional vocabulary (part numbers, process jargon) you must build your own dictionary; (2) how the Chinese word segmenters (jieba and friends) actually work under the hood, and each one's pros and cons; (3) how one query flows from tokenization → BM25 retrieval-and-ranking → dense-vector ranking → RRF fusion — which documents each lane retrieves, how each ranks them, and how to set the parameters — traced end-to-end on a concrete example. And when covering BM25 I'll start from the bottom: how term frequency is counted and why mid-to-low-frequency terms are the useful ones.
Why professional vocabulary demands a custom dictionary
Retrieval's first step is splitting text into tokens, and a token is the matching unit for BM25 downstream and the input unit for the embedding model. The problem: a generic segmenter has never seen your domain terms, so it splits them wrong.
query: "0402 貼片電阻 開路 不良" (0402 chip-resistor open-circuit defect)
without a custom dict (jieba default):
0402 / 貼片 / 電阻 / 開路 / 不良 ← "chip-resistor" split into two words
sometimes even: 貼 / 片 / 電阻 ← more shattered
with a custom dict (chip-resistor, open-circuit, 0402 as single words):
0402 / 貼片電阻 / 開路 / 不良 ← the domain term becomes one atomic token
The damage from shattering is twofold. The BM25 lane matches on tokens; once "chip-resistor" splits into "chip" + "resistor," a document only about "chip tape/adhesive" gets retrieved just for matching "chip" — precision drops. The dense lane feeds the model these shattered fragments too, diluting the semantics. So the first thing in domain retrieval is often not swapping models, but adding part numbers, process abbreviations, and model codes to the dictionary so they stay whole.
Tokenizer family (1): the segmenters — they use your dictionary
This family outputs actual "words," deciding splits via a dictionary (or a model on top of a dictionary), and you can feed it a custom dictionary. The workhorses of Chinese retrieval live here.
jieba — down to the bottom
jieba's accurate mode is four steps, each worth seeing clearly [1]:
1. Prefix dictionary: load the dict into a "prefix index" (O(1) test of whether a
prefix starts some word)
2. Build a DAG: scan the sentence, for each start list "all possible ends in the dict,"
connect into a directed acyclic graph
"貼片電阻" → possible splits: 貼|片|電|阻, 貼片|電阻, 貼片電阻… all edges on the graph
3. Dynamic programming for the max-probability path: each dict word has a frequency → probability;
DP from sentence end to start computes "which split path has the highest total probability"
4. Out-of-vocabulary (OOV): words not in the dict use an HMM (label each char as B/E/M/S position)
+ the Viterbi algorithm to infer the most likely label sequence and regroup chars into words
Three modes: accurate (the above, for retrieval and analysis), full (emit every possible word combination — fast but ambiguous), search-engine (accurate mode, then further cut long words into short ones to raise recall). Custom dictionary is direct: jieba.load_userdict(), one line per "word freq pos"; raise the frequency and that word more easily wins the DP over the default split (or force it with suggest_freq).
- Pros: light, fast, pure Python, custom dict, offline.
- Cons: trained mostly on Simplified Chinese; Traditional Chinese and new words lean on the HMM with limited power; no deep context (weak on same-form ambiguity).
CKIP (Academia Sinica) — deep-learning sequence labeling
CKIP transformers [8] do segmentation + POS + NER with BERT-class models, most accurate for Traditional Chinese, great for Taiwanese professional text. Cons: heavy, slow, wants a GPU; custom vocabulary usually needs fine-tuning or post-processing, unlike jieba where you just drop a txt.
IK Analyzer (for Elasticsearch)
If your BM25 runs on Elasticsearch, IK is standard [7]. Two modes: ik_max_word (fine-grained, exhaustively enumerates possible words — used at index time to raise recall) and ik_smart (coarse, only the single most reasonable split — used at query time). Custom dictionaries live in IKAnalyzer.cfg.xml with hot reload (add words without rebuilding the index).
pkuseg / THULAC / HanLP
pkuseg [6] sells multi-domain — separate models trained for news, medicine, web, etc., and it can be retrained on your own domain data, beating generic segmenters on verticals like manufacturing. THULAC is fast with POS; HanLP is feature-complete (multilingual, multi-task).
Segmenter cheat sheet
┌──────────┬──────────────┬───────────────┬──────────────────────────┐
│ tool │ under hood │ custom dict │ role / trade-off │
├──────────┼──────────────┼───────────────┼──────────────────────────┤
│ jieba │ dict+DP+HMM │ load_userdict │ light/fast; Simp strong │
│ CKIP │ BERT labeling│ fine-tune/post│ Trad best; heavy; GPU │
│ IK(ES) │ dict enumerate│ cfg.xml hot │ lives in ES; feeds BM25 │
│ pkuseg │ domain models│ retrain domain│ multi-domain; verticals │
│ THULAC │ struct. pred.│ limited │ fast; with POS │
└──────────┴──────────────┴───────────────┴──────────────────────────┘
Tokenizer family (2): the subword camp — it ignores your dictionary
This camp is statistical subwords for neural models (embeddings / LLMs). It doesn't read your dictionary; a fixed vocabulary splits any string into subwords.
- BPE [3]: start from characters and repeatedly merge the most frequent adjacent pair (post #4).
- WordPiece (BERT): merge the pair that most increases corpus likelihood, not simply the most frequent.
- Unigram LM [4]: the reverse — start with a large subword vocabulary and iteratively drop the subwords whose removal least hurts overall likelihood; it can emit multiple splits with probabilities for the same string (subword regularization). SentencePiece [5] consumes raw strings, language-independent, with both BPE and Unigram built in.
Traits: fixed vocabulary, never OOV, but shatters domain terms — RC0402 may become RC/04/02. That's the fundamental division of labor with the segmenter camp:
The fundamental difference
┌────────────┬─────────────────────┬─────────────────────────┐
│ │ segmenters(jieba/IK)│ subword(BPE/Unigram) │
├────────────┼─────────────────────┼─────────────────────────┤
│ output │ real "words" │ statistical subwords │
│ uses dict? │ yes(add domain terms)│ no(fixed vocab) │
│ OOV │ yes(HMM patches it) │ none(splits to chars) │
│ used by │ BM25/keyword search │ embedding/LLM models │
│ domain term│ kept whole via dict │ often shattered(finetune)│
└────────────┴─────────────────────┴─────────────────────────┘
Bottom line: the BM25 (sparse) lane uses a segmenter + your custom dictionary; the embedding (dense) lane uses the model's own subword tokenizer. The two lanes understand "words" differently — which is exactly why hybrid retrieval must fuse them.
BM25 from the "term frequency" details up (answering "mid-to-low frequency")
BM25 matches the terms the tokenizer produced, so how TF (term frequency) is counted is entirely decided by the split. TF is just "how many times this term appears in this document" — but the key point: not every word is equally useful.
This goes back to Luhn's 1958 insight [2]: a word's resolving power isn't "higher frequency is better" — it's strongest in the mid band.
Luhn's frequency-vs-resolving-power (upper & lower cutoffs)
resolving
power
│ ______
│ / \ ← mid-to-low freq: highest power (significant words)
│ / \
│______/ \______
│ ↑ ↑
└──┴─────────────────────────┴──────▶ frequency (sorted high→low)
upper cutoff lower cutoff
high-freq (的/是/"defect") very-low-freq (typos / one-off noise)
everywhere → no power too rare → mostly noise
In plain terms: high-frequency words (the, is, or even "defect" if it's everywhere in your corpus) appear in nearly every doc and can't tell them apart — cut by the upper cutoff; very-low-frequency words (typos, one-off noise) are too rare to represent anything — cut by the lower cutoff; mid-to-low-frequency words (0402, chip-resistor, open-circuit) are just right — common enough to match, rare enough to discriminate.
IDF (inverse document frequency) is that intuition made math: IDF = log(N / df), where df is the number of documents containing the term. High-freq words have large df → IDF near 0; mid-to-low-freq words have small df → high IDF. So a BM25 score = Σ (each query term's IDF × that term's TF contribution), inherently letting mid-to-low-frequency terms drive the ranking.
BM25 then adds two corrections (formula dissected in post #4; here the intuition): TF saturation (k1) — a term appearing 20 times shouldn't count 10× two times, so it saturates toward a ceiling; length normalization (b) — a long document shouldn't win just for having more words. Starting parameters: k1=1.2~2.0, b=0.75. For short part-number descriptions (short fields) you can lower b (length penalty is meaningless); for long spec sheets keep b=0.75.
The full walkthrough: tracing one query's ranking
Now string it all together. Setup:
query: "0402 貼片電阻 開路 不良"
custom dict sets "chip-resistor, open-circuit, 0402" as whole words
candidate corpus (simplified to 5 work-order / knowledge docs):
D1: 0402 chip-resistor open-circuit causing whole-batch defect analysis
D2: chip-resistor solder void causing open-circuit
D3: 0603 capacitor short-circuit defect case
D4: 0402 resistor rated-power derating curve
D5: open-circuit inspection method and instruments
Step 0 — Tokenizer: jieba + userdict splits the query into [0402, chip-resistor, open-circuit, defect]; each doc is split the same way. (Without a custom dict, chip-resistor scatters into chip/resistor, and any doc with an unrelated "chip" term would falsely match.)
Step 1 — BM25 sparse retrieval: compute IDF for the query's 4 terms, then BM25 per doc. defect appears in D1, D3 → high df → low IDF (higher-frequency, weak power); chip-resistor, open-circuit, 0402 have low df → high IDF (mid-to-low freq, strong power). Accumulate per doc:
BM25 hits (a ✓ contributes that term's IDF×TF)
0402 chip-res open-cir defect → rank A
D1 ✓ ✓ ✓ ✓ 1 (all four hit, highest)
D2 ✓ ✓ 2 (two high-IDF terms)
D4 ✓ 3 (only 0402)
D5 ✓ 4 (only open-circuit)
D3 ✓ 5 (only the low-IDF "defect")
Step 2 — dense / MaxSim semantic retrieval: embed the whole query, compare to each doc's vector. This lane reads semantic intent, not surface tokens: D2 "solder void causing open-circuit" matches "the cause of open-circuit defects" best semantically even with few literal hits; D5 "open-circuit inspection" is semantically related; D4 has 0402 but is about derating → intent differs → lower.
dense similarity ranking → rank B
D2 1 (best semantic fit for "cause of open-circuit")
D5 2 (open-circuit inspection, related)
D1 3 (relevant but diluted by "whole-batch analysis")
D4 4 (0402 but topic is derating)
D3 5 (capacitor short-circuit, off-topic)
Step 3 — RRF fusion: the two lanes have different scales (BM25 is an unbounded real, cosine is −1~1), so don't add — use rank-only RRF, k=60, score = Σ 1/(k+rank):
rank A rank B RRF = 1/(60+rA) + 1/(60+rB)
D1 1 3 1/61 + 1/63 = 0.01639 + 0.01587 = 0.03227
D2 2 1 1/62 + 1/61 = 0.01613 + 0.01639 = 0.03252 ← highest
D5 4 2 1/64 + 1/62 = 0.01563 + 0.01613 = 0.03175
D4 3 4 1/63 + 1/64 = 0.01587 + 0.01563 = 0.03150
D3 5 5 1/65 + 1/65 = 0.01538 + 0.01538 = 0.03077
final order: D2 > D1 > D5 > D4 > D3
Read this table and you get the value of fusion: BM25 alone ranks the literally-complete D1 first; but RRF lifts D2 to first — because D2 is #2 in sparse and #1 in dense, strong in both lanes, more robust than D1 which only tops one. D5, merely mid in each lane, still beats D4, again because it's decent in both. That's the spirit of hybrid: only what convinces both judges — keyword and semantics — is truly relevant.
Parameters (practical starting point): each lane retrieves top-50 candidates → fuse with RRF (k=60) → take fused top-10 for the next stage (cross-encoder rerank or straight to the LLM). BM25: k1=1.2, b=0.75. Retrieve too few and you miss (recall gap); too many and RRF's tail adds noise and slows the reranker.
Why RRF is designed this way (a bit deeper)
RRF's key is using only ranks, not raw scores [10]. Why? BM25 scores and cosine have no comparable scale, so a weighted sum (α·BM25 + β·cosine) requires normalizing each first, and normalization is unstable across queries (BM25's range drifts with query length). RRF sidesteps this — rank is scale-free.
The shape of 1/(k+rank): k=60 acts as a smoothing factor, making #1 (1/61≈0.0164) and #2 (1/62≈0.0161) close but still ordered, and giving mid/low ranks (#50 → 1/110≈0.0091) a non-zero contribution — so a doc "mid in both lanes" can climb by accumulation. Smaller k → weight the head more (bigger head-tail gap); larger k → flatter (everyone closer). Conversely, weighted-score fusion is interpretable and lets you finely tune each lane's weight, but must maintain normalization and is scale-sensitive — so production usually uses RRF as a robust default and switches to weighting only with a clear reason.
Landing it in your pgvector / ES part-number retrieval
Mapping the whole chain to your stack:
text ─[segmenter + custom dict]─┬─[BM25 sparse]───┐
(add part#/process terms │ │
to userdict) │ │
─[subword tokenizer]────────└─[embedding]────┤
▼
[RRF fusion k=60]
▼
top-10 →(optional rerank)→ LLM
Concretely: sparse lane — on Elasticsearch, index with ik_max_word + custom dict, query with ik_smart; on Postgres, use built-in full-text search or a BM25 extension and feed it your part-number dictionary. Dense lane — pgvector's <=> (cosine). Fusion — RRF on the two rankings in the app layer. Dictionary maintenance is the long-term chore: keep adding new part numbers and process abbreviations to the userdict / IK dict (IK hot-reloads; jieba uses add_word / frequency tweaks). Starting parameters: k1=1.2, b=0.75; RRF k=60; top-50 per lane → fuse top-10.
Closing the loop with one causal chain
The tokenizer decides BM25's matching unit and the embedding's input unit — split it wrong and everything skews → domain vocabulary must stay whole via a custom dictionary (segmenters jieba/IK/pkuseg use dicts; subword BPE/Unigram don't and shatter terms, for models) → BM25's TF is decided by the split, and the truly discriminative terms are mid-to-low frequency (Luhn's resolving power), which IDF formalizes, with k1/b correcting saturation and length → sparse (literal) and dense (semantic) each produce a ranking → different scales, so fuse with rank-only RRF, where being high in both lanes wins → parameters: top-50 per lane, RRF k=60, fuse top-10.
In one line: retrieval quality is decided at tokenization; mind your domain dictionary, let mid-to-low-frequency terms drive BM25, then use RRF so literal and semantic evidence cross-validate — that's robust hybrid retrieval.
Extension hooks
- Effect of jieba's HMM switch: turning HMM off (
HMM=False) can be more stable when all your part numbers are known — when to turn it off. - Operating IK's hot-reload dictionary: how to mount a remote dict file and how often it reloads.
- Learned sparse (SPLADE): let a model learn "which term matters," skipping the manual dictionary and IDF (continues posts #2, #4).
- Weighted RRF variants: when weighting one lane (weighted RRF) is worth it.
- The granularity-vs-recall/precision curve:
ik_max_wordvsik_smartmeasured on your part-number corpus.
Tell me which to go deep on.
References
[1] jieba Chinese segmentation (fxsjy/jieba) — prefix dict + DAG + dynamic programming + HMM/Viterbi. — https://github.com/fxsjy/jieba
[2] Luhn, H. P. (1958). The Automatic Creation of Literature Abstracts (frequency resolving power / cutoffs). IBM J. R&D. — https://doi.org/10.1147/rd.22.0159
[3] Sennrich, R. et al. (2016). Neural Machine Translation of Rare Words with Subword Units (BPE). — https://arxiv.org/abs/1508.07909
[4] Kudo, T. (2018). Subword Regularization: Improving NMT Models with Multiple Subword Candidates (Unigram LM). — https://arxiv.org/abs/1804.10959
[5] Kudo, T., Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer. — https://arxiv.org/abs/1808.06226
[6] Luo, R. et al. (2019). PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation. — https://arxiv.org/abs/1906.11455
[7] IK Analysis plugin for Elasticsearch/OpenSearch (ik_smart / ik_max_word, hot-reloadable custom dict). — https://github.com/infinilabs/analysis-ik
[8] CKIP Transformers (Traditional Chinese segmentation / POS / NER). — https://github.com/ckiplab/ckip-transformers
[9] Robertson, S., Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. — https://doi.org/10.1561/1500000019
[10] Cormack, G. et al. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR. — https://doi.org/10.1145/1571941.1572114
[11] Khattab, O., Zaharia, M. (2020). ColBERT: Contextualized Late Interaction over BERT (MaxSim). — https://arxiv.org/abs/2004.12832
[12] Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering (DPR). — https://arxiv.org/abs/2004.04906