Following on from post #5: four advanced topics, taken all the way down
The first five posts built the backbone of RAG retrieval. This one drills into four hands-on advanced topics, deliberately more detailed than before — each covered as "what it solves, how it works step by step under the hood, how it compares, and how to set it in practice": SPLADE learned sparse, chunking strategies, weighted RRF, and IK dictionary hot-reload ops.
1. SPLADE: learned sparse — a neural upgrade to BM25
What it solves
A tension recurs through posts #2/#4/#5: BM25 (sparse) is exact but has no semantics (a user says "NG," the doc says "defect," and they don't match — vocabulary mismatch); dense is semantic but a black box needing a separate ANN index. SPLADE [1] wants both: it outputs a sparse vector (dimension = vocabulary size, fits an inverted index like BM25), but the weight of each term, and whether to "expand" to related terms not in the text, are learned by a model.
The mechanism (step by step)
The key is reusing BERT's MLM head — the layer that already maps each position to "logits over the whole vocabulary (~30k WordPiece terms)." SPLADE uses it like this:
input text ── BERT ──▶ every token position i has a |V|(~30000)-dim logit row
(MLM head: which vocab terms this position "looks like")
for each term j in the vocabulary, compute its weight:
w_j = max over positions i of log(1 + ReLU(logit_{i,j}))
└ ReLU: clamp negatives to 0 (keep only positive contributions)
└ log(1+·): saturation, so no single big value dominates (cf. BM25 TF saturation)
└ max pooling: the strongest signal for term j across the whole text
result: a |V|-dim vector that is mostly 0 (sparse)
non-zero entries include terms NOT in the text but semantically related → term expansion
e.g. a doc saying "open-circuit" may also activate "disconnect / fault / solder / open"
To keep it sparse enough for an inverted index, training adds a FLOPS regularizer (an L1-like penalty that directly lowers the average number of non-zeros). Similarity is then almost identical to BM25 — query sparse vector · doc sparse vector, multiplying only over terms non-zero on both sides — the only differences: the weights are learned, and both sides are semantically expanded, so it hits docs that are "semantically relevant but literally absent."
Training and variants
SPLADE trains with the same recipe as dense retrievers: contrastive learning + hard negatives + distillation from a cross-encoder (SPLADE++ / "From Distillation to Hard Negative Sampling" [3], strong out-of-domain on BEIR). SPLADE v2 [2] switches pooling from sum to max and proposes a doc-only expansion variant — the query side skips the model, dropping online latency sharply (a very practical trade-off).
Compared with BM25 / dense / ColBERT
┌────────────┬──────────────┬───────────────┬───────────┬──────────┬──────────┐
│ │ representation│ index │ semantic │ interpret│ cost │
│ │ │ │ expansion │ -able │ │
├────────────┼──────────────┼───────────────┼───────────┼──────────┼──────────┤
│ BM25 │ sparse(stat) │ inverted │ none │ high │ tiny │
│ SPLADE │ sparse(learn)│ inverted │ yes(learn)│ high │ mid(model)│
│ DPR(dense) │ dense 1-vec │ ANN(HNSW) │ implicit │ low(black)│ mid │
│ ColBERT │ multi/token │ special(token)│ yes │ mid │ high(store)│
└────────────┴──────────────┴───────────────┴───────────┴──────────┴──────────┘
Pros: interpretable (you can list "which terms fired and their weights"), keeps exact token hits (part numbers), plus semantic expansion, reuses Lucene/ES inverted-index infra, robust cross-domain. Cons: expansion inflates index size and query latency (more non-zero terms), the query side must run a model (unless doc-only), and it consumes WordPiece subwords — shattering RC0402 (cf. post #5: domain terms get split in the subword camp; this is SPLADE's innate weakness on part numbers, patched via expansion or fine-tuning).
2. Chunking strategies — the retrieval unit decides recall
Why it matters
The chunk is the atomic unit of retrieval: what you embed, what BM25 indexes, and what you finally feed the LLM are all chunks. Too big → one vector must represent too many topics, diluted, noisy; too small → loses context on its own. In practice, the split affects recall more than swapping the embedding model.
One by one (mechanism + trade-off)
- Fixed-size + overlap: hard-cut by character/token count, with N overlap between neighbors (so a sentence cut in the middle isn't lost on both sides). Simplest, fastest, but often cuts mid-sentence/semantic-unit.
- Recursive character splitting: split by a hierarchy of separators — try
\n\n(paragraph), then\n(line),.(sentence), space — breaking at the most natural semantic boundary while respecting the size cap. LangChain's default; great value. - Semantic chunking: embed sentence by sentence, scan the similarity between neighbors, and cut where it drops below a threshold (a topic shift). Boundaries hug semantics, but you first compute a pass of sentence vectors.
- Sentence-window: embed a single sentence (precise retrieval), but on a hit return that sentence + N neighbors as context — "retrieve small, feed large," balancing precision and completeness.
- Parent-document (hierarchical): index fine child chunks (good for matching), and on a hit return the parent (larger block) to the LLM. Same idea as sentence-window: decouple retrieval granularity from generation granularity.
- Propositions (Dense X Retrieval) [9]: use an LLM to break text into self-contained atomic facts as chunks (finest granularity) — high retrieval precision, at the cost of LLM preprocessing.
- RAPTOR (tree summaries) [7]: recursively embed → cluster (Gaussian Mixture) → LLM-summarize each cluster → go up a level, building a tree (leaves = source detail, higher = more abstract summaries). At query time retrieve across levels, getting both detail and global summary — especially strong for multi-hop questions that need to synthesize a whole long document.
- Late chunking [8]: counter-conventional — first embed all tokens of the whole document with a long-context model, then mean-pool the token vectors by chunk boundary into per-chunk vectors. Each chunk vector thus carries full-document context, solving the old "chunks embedded independently lose context" problem, and needs no extra training.
data flow of several splits
sentence-window: [sentence]←embed; on hit return [prev N | sentence | next N]
parent-document: index[small child]; on hit → return [large parent]
RAPTOR: leaf chunks→cluster→summarize→upper nodes→… (query across levels)
late chunking: whole doc→per-token embedding→mean-pool by boundary→context-aware chunk vecs
Comparison and parameters
┌──────────────┬──────────┬─────────┬───────────┬──────────────────────┐
│ strategy │ retrieval│ context │ preprocess│ fits │
│ │ granular.│ │ cost │ │
├──────────────┼──────────┼─────────┼───────────┼──────────────────────┤
│ fixed+overlap│ mid │ weak │ tiny │ starter, generic │
│ recursive │ mid │ mid │ low │ default(LangChain) │
│ semantic │ by topic │ mid │ mid(sent vecs)│ topic-mixed long docs│
│ sentence-win │ fine(ret)│ strong(ret)│ low │ precise hit + full ctx│
│ parent-doc │ fine/big │ strong │ low │ lists / spec sheets │
│ propositions │ finest │ weak→join│ high(LLM) │ factoid QA │
│ RAPTOR │ multi-lvl│ strongest│ high(clust+sum)│ whole-doc synthesis│
│ late chunking│ mid │ strong(full)│ mid(long-ctx)│ long docs, ctx+chunks│
└──────────────┴──────────┴─────────┴───────────┴──────────────────────┘
params: chunk_size (often 256~512 tokens), overlap (10~20%). Start with recursive;
use parent-doc or late chunking for long spec sheets; RAPTOR for whole-doc synthesis.
3. Weighted RRF vs other fusion methods
Plain RRF recap
From posts #4/#5: RRF(d)=Σ 1/(k+rank_r(d)), rank-only, normalization-free, k=60. Robust, tuning-free, the hybrid default.
Weighted RRF
When one lane is systematically more accurate (e.g., dense clearly beats sparse on your data), plain RRF's equal treatment loses out. The weighted version gives each lane a weight:
weighted RRF: score(d) = Σ w_r · 1/(k + rank_r(d)) (k may differ per lane)
lane r
e.g. dense more accurate → w_dense=1.5, w_sparse=1.0
cost: you lose plain RRF's tuning-free robustness; w must be tuned on a labeled dev set (overfit risk).
Compared with other fusion methods
┌────────────────┬──────────────┬────────┬────────┬────────────────────────┐
│ fusion │ normalize? │ tuning │ robust │ notes │
├────────────────┼──────────────┼────────┼────────┼────────────────────────┤
│ plain RRF │ no │ ~none │ high │ rank-only; top default │
│ weighted RRF │ no │ few(w) │ mid-hi │ ranks + per-lane weight│
│ weighted score │ yes(min-max/z)│ much │ low │ α·sparse+β·dense; scale-sensitive│
│ relative score │ yes(distrib.)│ mid │ mid │ normalize by score dist (Weaviate)│
│ learned(LTR) │ feature-dep │ most │ data-dep│ model-learned; needs training data│
└────────────────┴──────────────┴────────┴────────┴────────────────────────┘
Weighted score fusion (convex combination α·norm(sparse)+β·norm(dense)) is tunable and interpretable, but must normalize both lanes' scores first (min-max or z-score), and normalization drifts per query and is scale-sensitive. Relative score fusion (e.g., Weaviate) normalizes by the score distribution before weighting — in between. Learned fusion / LTR learns weights with a model — strongest but needs labeled data and training. How to tune weights: prepare labeled dev queries (which doc is relevant), use nDCG / recall@k as the metric, grid-search w (and k), pick the dev-best — and re-verify on an independent test set to avoid overfitting.
4. IK dictionary hot-reload ops — mechanism and pitfalls
Local vs remote dictionary
IK [11] mounts custom dictionaries two ways: local ext_dict (file on the node; a change requires a node restart) and remote remote_ext_dict (points to an HTTP URL; hot-reloadable, no restart). Production almost always uses remote.
The hot-reload mechanism
IK runs a Monitor background thread that, by default about every 60s, sends an HTTP
request to the remote_ext_dict URL:
IK ──HTTP GET/HEAD──▶ your dict service (returns .txt, one word per line, UTF-8 no BOM)
must return two headers: Last-Modified, ETag
IK compares these headers to the previous values:
either changed → dictionary updated → re-download the whole file → reload into memory
(swap the analyzer's word table)
neither changed → do nothing
So operationally you just: host the dict file at an HTTP endpoint that correctly returns Last-Modified and ETag; to add words, update that file (and have the server bump those headers); IK picks it up on the next poll and reloads — no node restart at all.
The key pitfall (must-know ops)
Hot reload swaps the in-memory analyzer, but analysis happens at two times: index time and query time. This creates the most common trap:
after adding a new word "chip-resistor" —
✔ query analysis: uses the new word immediately (next query works)
✔ documents indexed afterward: split and indexed with the new word
✘ already-indexed old documents: their tokens were split by the OLD dictionary and
frozen in the inverted index
→ not re-split automatically → old data still won't
benefit from the new word
Conclusion: to make OLD data honor the new word, you must reindex those documents.
Hot reload saves the "node restart," not the "reindex."
This is the root cause of "I added a part-number word, but search only finds new data, not old part numbers." Practice: hot reload makes "query side + new data" take effect instantly; then periodically / batch reindex to align historical data.
Compared with jieba / pgvector
jieba's add_word / load_userdict take effect at runtime, but each Python process loads its own — not distributed, so multiple workers must each sync the dict file. pgvector has no built-in Chinese-segmentation hot reload (segmentation happens in the app layer before insert), so you manage the dictionary in the app yourself. Of the three, IK's "remote dict + polling hot reload" is the most operationally mature.
Landing it in your part-number retrieval: composing the four
text ─┬─[SPLADE or BM25(IK+remote dict)]─┐
└─[dense embedding]────────────────┤
(long spec sheets: parent-doc / late chunking)
▼
[weighted RRF: up-weight dense if it wins on your data]
▼
top-k →(cross-encoder rerank)→ LLM
dictionary: put part-number/process terms in IK's remote dict (hot reload) + periodic reindex to align history
Closing the loop with one causal chain
SPLADE uses BERT's MLM head to turn text into a "learned sparse vector" (ReLU + log saturation + max pooling + FLOPS regularization), getting exact hits, semantic expansion, and an inverted index at once → but the atomic unit of retrieval is the chunk, and the split (recursive/sentence-window/parent-doc/RAPTOR/late chunking) decides recall, chosen by document type → fuse multiple lanes with RRF; when one lane is systematically better switch to weighted RRF, but tune the weights on a dev set and accept the tuning risk → for the Chinese lane, IK's remote hot-reload makes new words take effect instantly, but already-indexed old data must be reindexed to align.
In one line: every advanced-retrieval slot is a "spend a little cost for a little quality" trade-off — SPLADE buys interpretable semantic sparsity, chunking buys recall, weighted RRF buys a ranking fitted to your data, IK hot-reload buys restart-free dictionary ops; knowing the internals is how you know how far to turn each knob.
Extension hooks
- Controlling SPLADE's index bloat: how to tune the FLOPS regularizer strength λ, the expansion-count vs latency curve.
- RAPTOR's tree depth and summary cost: how deep pays off, which model tier to summarize with.
- Late chunking's long-context ceiling: how to segment-then-stitch beyond the model's context.
- Auto-tuning weighted RRF: use online click feedback (implicit labels) to keep updating the weights.
- IK reindex strategy: zero-downtime rebuilds via alias switching (alias + reindex).
Tell me which to go deep on.
References
[1] Formal, T. et al. (2021). SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. — https://arxiv.org/abs/2107.05720
[2] Formal, T. et al. (2021). SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. — https://arxiv.org/abs/2109.10086
[3] Formal, T. et al. (2022). From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective (SPLADE++). — https://arxiv.org/abs/2205.04733
[4] Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers (MLM head). — https://arxiv.org/abs/1810.04805
[5] Robertson, S., Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. — https://doi.org/10.1561/1500000019
[6] Khattab, O., Zaharia, M. (2020). ColBERT: Contextualized Late Interaction over BERT. — https://arxiv.org/abs/2004.12832
[7] Sarthi, P. et al. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. — https://arxiv.org/abs/2401.18059
[8] Günther, M. et al. (2024). Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. — https://arxiv.org/abs/2409.04701
[9] Chen, T. et al. (2023). Dense X Retrieval: What Retrieval Granularity Should We Use? (propositions). — https://arxiv.org/abs/2312.06648
[10] Cormack, G. et al. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR. — https://doi.org/10.1145/1571941.1572114
[11] IK Analysis plugin for Elasticsearch/OpenSearch (remote_ext_dict hot reload). — https://github.com/infinilabs/analysis-ik
[12] Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering (DPR). — https://arxiv.org/abs/2004.04906