Following on from last time: the question shifts from "whether to embed" to "which embedding"
Last post laid the foundation: an embedding is useful not because it's a matrix, but because a training objective carves the vector space so that "geometric distance = semantic distance." That post also planted a thread — compressing a text into a single vector inherently discards fine-grained token interaction, so pure embeddings (bi-encoders) have a precision ceiling and need a cross-encoder reranker to close the gap.
This post tackles the real next question in practice: BGE-M3, multilingual-e5, NV-Embed, OpenAI text-embedding-3, Cohere, Jina, Qwen3-Embedding… with this many models, what actually differs, and which should you pick for part-number retrieval? I won't recite spec sheets brand by brand — that's a catalog. We'll spread them across three "bottom-level axes"; each model is just a coordinate on those three axes. Understand the axes and you can place any new model yourself.
Divide 1: output form — dense / sparse / multi-vector
This is the deepest and most overlooked difference. Every embedding model turns text into "something comparable," but what shape it turns into splits into three camps, which directly decide how it's indexed, how similarity is scored, and where precision and cost land.
Dense (a single dense vector) is last post's protagonist: a whole text → one d-dimensional dense vector, compared by cosine. Its similarity is a single comparison of "overall meaning" — precomputable, blazing fast over the whole DB via ANN. The downside: detail gets averaged out — in a part number like RC0402FR-0710KL, that crucial 0710KL gets diluted when the whole string is squeezed into one vector.
Sparse / learned lexical (learned sparse term weights) takes another road. It outputs not a dense vector but a sparse vector whose dimension equals the vocabulary size, mostly zeros — each non-zero slot corresponds to an actual term, and its value is a learned weight for "how important this term is in this text." The landmark is SPLADE [3], which borrows BERT's MLM head to "expand" each token over the whole vocabulary with sparsity regularization, keeping classic keyword retrieval's virtues (exact term matching, inverted-index efficiency) while adding model-learned weights and synonym expansion. This matters for your use case: sparse is inherently great at exact token hits — part numbers, process abbreviations, spec digits, where one character off means a different thing, are often more reliably matched by sparse than dense.
Multi-vector / late interaction is the middle ground between bi-encoder and cross-encoder, and it pays off last post's thread. The landmark is ColBERT [2] (Contextualized Late Interaction over BERT). Instead of squeezing a document into one vector, it keeps a vector for every token; at scoring time it uses MaxSim: each query token finds its best-matching token in the document, and those maxima are summed.
How the three output forms score similarity
dense (single vec) sparse (term weights) multi-vector (ColBERT / MaxSim)
q -> [·······] q -> {resistor:1.8, q -> [t1][t2][t3] (one vec per token)
d -> [·······] 0402:1.2, ...} d -> [t1][t2]...[tn]
cos(q,d) d -> {resistor:1.5, score = Σ max cos(q_i, d_j)
one holistic compare ohm:0.9, ...} i j
score = Σ matched term wts each query token finds its best match
fast, detail averaged exact token hits, inverted keeps token-level interaction; high
index friendly precision; must store n vectors
The mechanism dictates their roles: dense is cheapest, fastest, scans the whole DB; sparse excels at exact hits and plugs into existing inverted-index infrastructure; multi-vector, by keeping token-level interaction, approaches cross-encoder precision while still precomputing the document-side token vectors (unlike a cross-encoder, which voids everything when the query changes) — at the cost of storage and compute an order of magnitude above a single vector. These three aren't replacements for one another; they're three points on one precision/cost spectrum.
Divide 2: backbone — encoder or decoder-only LLM
The second axis is "what Transformer sits underneath." It decides how much world knowledge the model packs, and what surgery is needed to make it an embedding model.
Encoder camp (BERT-style, bidirectional): models like BGE and multilingual-e5-base are built on bidirectional encoders. Every token natively sees the full left and right context — great for producing "whole-passage understanding" representations, small and fast, the workhorse of retrieval for years.
Decoder-LLM camp: retrofit a decoder-only large language model (Mistral, Qwen, etc.) into an embedding model — e.g. E5-mistral [6], NV-Embed [7], Qwen3-Embedding [10]. The motivation is direct: a big LLM's world knowledge and semantic parsing, learned on massive corpora, far exceed a small encoder's. But using it as an embedder has two innate obstacles — which pay off two concepts from last post:
First, the causal mask. A decoder LLM is trained so each token sees only its left (for left-to-right generation), which is crippled for "understanding a whole sentence." NV-Embed's fix is to drop the causal mask during training and go bidirectional, which empirically beats keeping it across MTEB [7]. Second, how to pool a sentence vector. Last post covered the problems with mean / [CLS]; decoder LLMs often use "the last token," also imperfect. NV-Embed proposes a latent attention layer (512 latents, 8 heads) dedicated to pooling, beating mean or last-token pooling [7]. Flag the layer idea from last post again: the backbone is a mechanism part; whether it makes a good embedding depends on pooling and objective chiseling it correctly.
Be honest about the cost too: a decoder-LLM embedder is often 7B params outputting 4096 dims — big index, slow inference; a small encoder (e5-base, bge-base) is frequently more cost-effective in real retrieval. "Bigger is more accurate" is not always worth it in embeddings.
Divide 3: training recipe — weak-supervision contrastive, hard negatives, instructions, self-distillation
Same backbone, different recipe, different product space (again, last post's "the product layer is decided by the loss"). Modern strong models roughly share these ingredients:
Two-stage contrastive training. Take E5 [4]: stage one does contrastive pretraining on ~1 billion "weakly supervised" text pairs (Wikipedia title↔passage, Reddit post↔reply, StackExchange question↔answer, and other naturally paired data); stage two fine-tunes on labeled high-quality data with hard negatives (distractors that are semantically close but actually irrelevant). Hard negatives are the precision key — with only random negatives, the model never learns to tell "close but wrong" apart.
Instruction prefixes. E5 requires prefixes like query: / passage:; instruction-tuned variants go further with a natural-language instruction (e.g. Represent this sentence for retrieval:) to steer one model toward different tasks [4][6]. Mechanically, that prefix influences every token's representation via self-attention — using text to "nudge" where the vector lands in space.
Self-knowledge distillation — BGE-M3's signature, detailed next.
The flagship of this wave: why BGE-M3 is special
Stack the three axes and you see why BGE-M3 [1] is this generation's emblematic model. Its name, M3, unpacks into three "multis":
- Multi-Linguality: 100+ languages, mixed Chinese/English and cross-lingual retrieval — a direct fit for your part-number descriptions that often mix scripts.
- Multi-Functionality: the fiercest point — the same model, in one forward pass, outputs all three forms: dense, sparse, and multi-vector. The three camps from Divide 1, all at once.
- Multi-Granularity: from short sentences to documents up to 8192 tokens — work orders, spec sheets, long passages without forced chunking.
How does it train the three forms together without them fighting? Via self-knowledge distillation: the relevance scores from the three retrieval heads (dense/sparse/multi-vector) are integrated into one "teacher" signal, which then teaches all three heads. The three forms tutor each other — dense learns multi-vector's finesse, sparse learns dense's semantics, strengths cross-pollinate [1].
For your MES / part-number retrieval, BGE-M3's practical value is hybrid retrieval from a single model: the sparse path handles "exact hits on tokens like 0710KL," the dense path handles "10KΩ resistor and 10K ohm resistor landing close in meaning," and fusing both scores plugs the two holes — pure dense missing exact part numbers, pure keyword missing semantics.
Deployment axes: Matryoshka, long context, task LoRA
A few "engineering knobs" for selection, all mentioned in or extending from last post:
Matryoshka truncatable dimensions [8]: OpenAI text-embedding-3 [11] (large 3072 dims by default, small 1536), Cohere, Jina, Qwen3 all support it — trained so "the leading dimensions form a good embedding on their own," so you can truncate 3072 to 256 or 1024 for pgvector, saving storage and comparison cost at a small accuracy loss. OpenAI's own number: te3-large truncated to 256 still beats the old ada-002 at its full 1536 [11].
Long context: Jina v3 [9] and BGE-M3 reach 8192 tokens — long work orders / specs without shredding.
Task LoRA: Jina v3's [9] trick — one shared backbone with multiple task-specific LoRA adapters (retrieval, clustering, classification, matching), switched by task at inference, trading tiny extra params for "one model, many tasks" flexibility.
One table placing them on the axes
┌─────────────────────┬───────────────┬────────────────────────┬──────────┬──────────┬──────┬─────────────────────┐
│ Model │ backbone │ output form │ dims │ max tok │ multi│ signature │
├─────────────────────┼───────────────┼────────────────────────┼──────────┼──────────┼──────┼─────────────────────┤
│ BGE-M3 │ encoder(XLM-R)│ dense+sparse+multi-vec │ 1024 │ 8192 │ 100+ │ 3 forms at once/self-KD│
│ multilingual-e5 │ encoder │ dense │ 1024 │ 512 │ yes │ weak-sup contrastive │
│ E5-mistral-7b │ decoder-LLM │ dense │ 4096 │ 32k │ ~ │ LLM as backbone │
│ NV-Embed-v2 │ decoder-LLM │ dense │ 4096 │ 32k │ ~ │ no causal mask+latent│
│ Qwen3-Embedding-8B │ decoder-LLM │ dense │ variable │ 32k │ yes │ top of MTEB multiling│
│ OpenAI te3-large │ proprietary │ dense (Matryoshka) │ ≤3072 │ 8191 │ yes │ truncatable/easy API │
│ Cohere embed v4 │ proprietary │ dense (Matryoshka) │ variable │ long │ yes │ enterprise/multimodal│
│ Jina embeddings v3 │ encoder │ dense (Matryoshka) │ ≤1024 │ 8192 │ yes │ task-specific LoRA │
└─────────────────────┴───────────────┴────────────────────────┴──────────┴──────────┴──────┴─────────────────────┘
Note: dims/lengths are approximate and vary by version/config; "signature" is each model's mechanistic tell.
How to choose for your MES / RAG: one decision chain
Turn selection into a few derivable judgments instead of copying whoever tops the leaderboard:
Multilingual / mixed script? Your part numbers and process notes mix Chinese and English → rule out English-only models; look at BGE-M3 / multilingual-e5 / Qwen3.
Need exact token hits? Part numbers and spec digits where one character means a different part → you need the sparse path → BGE-M3 hybrid (dense+sparse fusion) is the simplest single-model answer; or a dense model + separate SPLADE/BM25.
How long are the docs? Long work orders, spec sheets → pick the 8192-token tier (BGE-M3 / Jina v3).
Compute and latency budget? Online, low-latency, high volume → small encoder (bge/e5-base) or truncated dense; reserve decoder-LLM or multi-vector for the offline fine-ranking stage.
Need peak precision? Don't expect a single embedding to top out — continuing last post's conclusion: embedding coarse filter (BGE-M3 hybrid works) → multi-vector or cross-encoder reranker for fine ranking; that two-stage pipeline is the precision/cost optimum. ColBERT's multi-vector is the "cheaper than cross-encoder, more accurate than pure dense" middle tier here.
Closing the loop
One causal chain: models are countless, but the real differences are three axes — output form (dense/sparse/multi-vector, deciding index and precision cost), backbone (encoder vs decoder-LLM, deciding knowledge and retrofit cost), and training recipe (contrastive + hard negatives + instructions + self-distillation, deciding product-space quality) → BGE-M3's significance is packing the three points of the "output form" axis into one model via self-distillation, plus multilingual and long context → for manufacturing part-number retrieval, dense+sparse hybrid plugs both the semantic and exact-hit holes → but the endgame of selection is still last post's line: embedding does coarse filtering, reranker does fine ranking; no matter how many models exist, they take their positions on that same pipeline.
Extension hooks
- Score fusion in hybrid retrieval: dense and sparse scores have different scales — how to normalize and weight (RRF vs weighted sum) so one doesn't drown the other.
- ColBERT's storage blowup: quantizing token-level vectors, pooling, PLAID indexing.
- Fine-tuning BGE-M3 on your part-number corpus: whether to touch the sparse head, and how to mine hard negatives from similar part numbers.
- Which MTEB subtask to read [12]: the Retrieval score is what you should watch — don't be misled by the overall average.
Tell me which one to go deep on.
References
[1] Chen, J. et al. (2024). BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. — https://arxiv.org/abs/2402.03216
[2] Khattab, O., Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. — https://arxiv.org/abs/2004.12832
[3] Formal, T. et al. (2021). SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. — https://arxiv.org/abs/2107.05720
[4] Wang, L. et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5). — https://arxiv.org/abs/2212.03533
[5] Wang, L. et al. (2024). Multilingual E5 Text Embeddings: A Technical Report. — https://arxiv.org/abs/2402.05672
[6] Wang, L. et al. (2024). Improving Text Embeddings with Large Language Models (E5-mistral). — https://arxiv.org/abs/2401.00368
[7] Lee, C. et al. (2024). NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. — https://arxiv.org/abs/2405.17428
[8] Kusupati, A. et al. (2022). Matryoshka Representation Learning. — https://arxiv.org/abs/2205.13147
[9] Sturua, S. et al. (2024). jina-embeddings-v3: Multilingual Embeddings With Task LoRA. — https://arxiv.org/abs/2409.10173
[10] Qwen Team (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. — https://qwenlm.github.io/blog/qwen3-embedding/
[11] OpenAI (2024). New embedding models and API updates (text-embedding-3). — https://openai.com/index/new-embedding-models-and-api-updates/
[12] Muennighoff, N. et al. (2022). MTEB: Massive Text Embedding Benchmark. — https://arxiv.org/abs/2210.07316