Start with one problem: how does a computer "read" a word?
Let's take the problem all the way down: there is no such thing as "text" inside a computer — only numbers. When you look up a part number RC0402FR-0710KL in Oracle, the database stores a string of bytes. But the moment you ask "what else is semantically close to this part?", bytes are useless — 10KL and 10KΩ are worlds apart as bytes yet nearly the same thing in meaning.
So every system that touches language is stuck with the same first step: turn a piece of text into a vector, such that "similar meaning" becomes "nearby vector." That sentence is the bedrock of this whole post. You'll object: "isn't that just matrices?" — yes, everything lands on matrices in the end. But the point was never whether it's a matrix; it's who put the numbers into the matrix, and what geometric shape they arranged them into. We'll go from the dumbest possible approach all the way to modern embedding models and make this precise.
First attempt: one-hot — it's a vector too, so where does it break?
Say the vocabulary has 50,000 words. The most intuitive idea is to number each word, then blow that number up into a vector that is "1 in exactly one slot, 0 everywhere else." This is one-hot encoding.
Vocabulary size V = 50000
"king" → [0, 0, ..., 1, ..., 0, 0] ← slot 8234 is 1
"queen" → [0, 0, ..., 0, ..., 1, 0] ← slot 41902 is 1
"resistor" → [0, 1, ..., 0, ..., 0, 0] ← slot 3 is 1
↑ every vector has length 50000, one non-zero slot
One-hot is a bona fide vector, and stacked up it's a matrix. But it's broken from the bottom in two ways.
First, dimensional blow-up and extreme sparsity. 50,000 words means 50,000 dimensions, almost all zeros per token — wasteful to store and compute, and it grows linearly with the vocabulary.
Second, and fatally: any two distinct words are orthogonal. Take the dot product of two one-hot vectors — unless it's the same word, the result is always 0, i.e. a 90° angle. In the one-hot world, the distance between king and queen is exactly the same as the distance between king and resistor. Meaning simply never entered the geometry.
cos(king, queen) = 0 ┐
cos(king, resistor) = 0 ├─ all zero, the model sees no closeness at all
cos(queen, banana) = 0 ┘
That's our first contrast set. It tells us: turning words into vectors is not the hard part; the hard part is making the geometric relationships between vectors carry semantic relationships. One-hot is a matrix — but a geometrically empty one.
The truth behind "it's all matrices": an embedding lookup is a matrix multiply
Now to your core question, head-on. The lowest-level step of an embedding model — turning a token into a dense vector — really is just a single matrix multiplication. No magic.
Prepare a weight matrix W of shape V × d (vocab × embedding dim, e.g. 50000 × 768). To get a word's embedding, multiply its one-hot vector by W:
one-hot(king) W (50000 × 768) embedding(king)
[0 … 1 … 0] × ┌───────────────────────────┐ = [0.21, -0.83, …, 0.05]
(1 × 50000) │ 0.11 0.90 … -0.04 │ row 0 (1 × 768)
│ … │
slot 8234 = 1 ▶│ 0.21 -0.83 … 0.05 │ row 8234 ◀── exactly this row
│ … │
└───────────────────────────┘ row 49999
Because one-hot has a single 1, the effect of this multiply is simply to pull out row 8234 of W, whole. That's why nn.Embedding doesn't actually multiply in practice — it just gathers the row by index. Identical result, far faster. And that's why it's called an embedding layer: it's one linear lookup layer.
So far, "it's all matrices" is entirely correct at the mechanism level: looking up an embedding is one-hot × W = grab a row of W.
But notice what just happened — the problem moved. One-hot has no semantics because we hand-numbered its vectors. And those 50000 × 768 numbers in W, if randomly initialized, give embeddings with no semantics either. So the question "what's special about an embedding model" has its real answer not in "matrix multiply," but in the next question: how do the numbers in W become meaningful?
What's special isn't the matrix — it's who places the numbers: a three-layer frame
This is the single most important cut in the post. When learning any system with a training component, you must separate three layers, or you'll credit the wrong thing:
┌───────────────────────┬───────────────────────────────┬──────────────────────────────┐
│ Layer │ What this layer is │ In embeddings, maps to │
├───────────────────────┼───────────────────────────────┼──────────────────────────────┤
│ Mechanism / │ the "parts": how information │ lookup (matrix multiply), │
│ architecture │ flows through the net │ self-attention, pooling │
│ Training objective │ what we *require* the net to │ skip-gram negative sampling, │
│ (loss) │ achieve │ MLM, InfoNCE contrastive loss│
│ Product │ the trained weights and the │ that W, and "cosine ≈ │
│ │ geometry they carve out │ semantic similarity" │
└───────────────────────┴───────────────────────────────┴──────────────────────────────┘
One line dissolves your question: "it's all matrices" describes the mechanism layer; what's special about an embedding model lives in the training-objective layer — some loss chisels the numbers in the matrix, bit by bit, into the shape "geometric distance = semantic distance." Matrix multiplies are everywhere; "every row of this matrix sits in its semantically correct place" is where the value is. The next three sections take "how the loss chisels this table" all the way down.
Mechanism 1: the distributional hypothesis and word2vec — turning meaning into direction
To make numbers mean something, you first need an assumption about where meaning comes from. Linguistics answers with the distributional hypothesis: a word's meaning is determined by the words that tend to appear around it. Firth (1957) put it best: "You shall know a word by the company it keeps." resistor and capacitor are close in meaning not by decree, but because they appear in extremely similar contexts (circuits, packaging, rated voltage…).
word2vec [1][2] turns that hypothesis into a training objective. In skip-gram, the task is: given a center word, predict the words in its surrounding window.
sentence: … 0402 package the [resistor] rated power …
center word
skip-gram demands: from the resistor vector,
we can predict its real neighbors "package / rated / power"
The real trick is negative sampling [2]. Each step, the model sees one true co-occurring (center, neighbor) positive pair, draws a few random words that almost certainly are not neighbors as negatives, and nudges the vectors so that:
positive: resistor · package → push dot product UP (pull directions together)
negative: resistor · banana → push dot product DOWN (push directions apart)
After hundreds of millions of such nudges, what happens? Words that keep showing up in similar contexts get squeezed into similar directions. Similar meaning → similar context → repeatedly pulled together by the same neighbors → close in geometry. That's how semantics grows into W.
The famous evidence is the linear analogy — differences in meaning become displacements in vector space:
queen ●
↖ (the +woman −man displacement)
king ●─────┘
│
└── vec(king) − vec(man) + vec(woman) ≈ vec(queen)
"king is to man as queen is to woman"
becomes a translatable segment in the space
Point out a layer relationship here: this "king−man+woman≈queen" ability is not given by the lookup mechanism — it's given by the skip-gram loss. Same matrix W, different objective, totally different geometry. GloVe [3] takes another road — factorizing a global co-occurrence-count matrix directly — but with the same goal: turn co-occurrence statistics into vector geometry. That's the first thing that's special about an embedding model.
Mechanism 2: from static to contextual — self-attention deforms the vector by context
word2vec has a hard limitation that lands right on a manufacturing pain point: it's static — one word, one vector, forever. But bank can be a riverbank or a financial bank; in your world, MOS could be a transistor or a line's Measurement System. A static embedding averages the two senses into one vector — wrong for both.
BERT [5] fixes exactly this: let the same word's vector change with context — a contextual embedding. The core part is the Transformer's [4] self-attention. Don't treat it as a black box — open it up and see what it computes.
Each token first gets an initial vector from the lookup, then projects three role vectors — Query (what I'm looking for), Key (what I can offer), Value (what I actually carry). Then:
for each token i, against every token j in the sentence:
1. relevance score: score(i,j) = Qᵢ · Kⱼ / √d
(dot product of i's question with j's answer;
dividing by √d keeps scores from exploding as d grows,
which would make softmax too spiky)
2. softmax normalize: α(i,·) = softmax(score(i,·))
(exponentiate then divide by the sum → a row of weights
summing to 1; the exp widens gaps so the most relevant j
gets the largest weight)
3. weighted sum: outᵢ = Σⱼ α(i,j) · Vⱼ
(i's new vector = a weighted average of the content of
the tokens it should attend to)
So the output vector of the bank token gets pulled in different directions depending on whether river or money is in the sentence — same word, different context, different vector. That's the root reason static loses to contextual: sense disambiguation is baked into the vector's computation by self-attention.
Flag the layers again: self-attention is a mechanism-layer part; what teaches it who to attend to is BERT's training objective MLM (Masked Language Modeling) — randomly mask words and make the model restore them. To guess the masked word, the network is forced to use context, and the attention weights get chiseled to "grab semantically related tokens." Once more: the ability comes from the loss, not the part itself.
Mechanism 3: why sentence vectors need extra training — anisotropy and contrastive learning
You might now think: to get a vector for a whole sentence (or a part-number description, or a work-order summary), can't I just average BERT's token outputs, or grab the [CLS] slot?
Empirically, that works badly, and the reason is deeply counterintuitive, worth its own section. Ethayarajh (2019) [7] found that raw BERT vectors are not spread evenly through space — they're all crammed into a narrow cone. This is anisotropy.
ideal (isotropic) reality (anisotropic)
vectors spread over the sphere vectors crammed into a narrow cone
↑ ● ↑ ●●●
● │ ● │ ●●●●● ← nearly the same direction
●────┼────● ● │ ●●●●●●
● │ ● │ ●●●
↓ ● └─────────→
angles between points are wide every pair has high cosine → can't tell apart
Why is this a disaster? Because if all vectors point roughly the same way, any two sentences report high cosine similarity, bunched together, with near-zero discriminative power. Try to "find the most similar work order" and it thinks every order looks alike.
Sentence-BERT (SBERT) [6] fixes this with a new objective that "pries open" the cone. It runs two sentences through the same BERT (weights shared — a siamese / two-tower structure), pools them into two sentence vectors, and trains with a contrastive objective: pull semantically equivalent sentences (positives) together, push unrelated ones (negatives) apart. The mathematical skeleton of such objectives is InfoNCE [8] (Info Noise-Contrastive Estimation):
exp( sim(a, b⁺) / τ )
L = − log ────────────────────────────────────
exp( sim(a,b⁺)/τ ) + Σⱼ exp( sim(a,bⱼ⁻)/τ )
a = anchor sentence vector
b⁺ = positive (same meaning) → numerator, make it large
bⱼ⁻ = a batch of negatives (unrelated)→ the other terms, make them small
sim = cosine similarity
τ = temperature, scales the scores; smaller τ punishes hard negatives harder
Read it apart: the loss maximizes the anchor–positive similarity as a share of the anchor's similarity to all candidates. In plain terms — among a pile of sentences, the positive must be the single nearest one to the anchor. After enough training, equivalent sentences are welded together and different ones pushed apart, and the narrow cone is pried into a usable space where "direction = meaning." SimCSE [9] goes further: even using "run the same sentence through the net twice, letting dropout make two slightly different versions" as a positive pair is enough to calibrate the space beautifully.
The key layer relationship of this section: BERT's mechanism (self-attention) does not by itself guarantee usable sentence vectors; it's the contrastive loss SBERT swaps in that carves the product space from a "narrow cone" into something "comparable." That's why a "sentence embedding model" is a thing you have to train on purpose, not something you get by casually pooling BERT.
Why almost everyone uses cosine similarity
We've used cosine throughout; here's the bottom-level reason. To measure closeness between two vectors, three options are common: Euclidean distance (straight-line distance between points), raw dot product (projection, amplified by vector length), and cosine (angle only, length normalized away).
Embedding retrieval almost always picks cosine, because the sim in the training objective (that InfoNCE line) is cosine — the product space encodes meaning along direction, while length mostly reflects noise like word frequency or sentence length. Looking only at direction, ignoring length, aligns exactly with how the space was carved.
cosine asks one thing: how nearly do the two arrows point the same way?
b
╱
╱ small θ → large cos → close in meaning
╱____ a
(how long a and b are doesn't matter; the angle θ does)
There's an engineering bonus: after L2-normalizing vectors (length scaled to 1), cosine equals the dot product, so you can score a whole batch with one fast matrix multiply — crucial when scanning millions of vectors.
Deriving strengths and weaknesses from the mechanism (not memorizing a list)
With the bottom layer in hand, the trade-offs can be derived rather than memorized:
Why it's fast and scalable: a text's embedding depends only on itself, not on whatever you'll later compare it against. So you can compute the whole knowledge base's vectors offline and store them (like your site's doc_chunks table); at query time you compute one vector and use approximate nearest neighbor (ANN) search over the stored vectors. "Store first, query later" turns the cost from "recompute every time" into "compute once, query many times."
Why it generalizes semantically: because meaning is encoded as direction, paraphrases that don't match verbatim (10KΩ vs 10K ohm resistor) can still land close — something one-hot / keyword matching can never do.
Cost 1, black box: what each of the 768 dimensions means is not human-interpretable; when it errs, it's hard to "see" why it judged two texts as close.
Cost 2, domain shift: the model's geometry is carved by its training corpus. A model trained on generic web text, dropped into your MES world of part numbers, process abbreviations, and mixed Chinese/English, may point semantics the wrong way — this is exactly the root of "why domain fine-tuning / instruction prefixes are needed."
Cost 3, static knowledge: a vector freezes the semantics of its training moment. New part numbers and new process jargon it never saw may not point correctly.
Complement vs conflict: bi-encoder vs cross-encoder, and its place in RAG
Finally, put it back into the full pipeline and use one contrast set to make clear who it can pair with, and why.
An embedding model is a classic bi-encoder: query and document are encoded into vectors separately, and combined only by one cosine at the end. Its opposite is the cross-encoder: it concatenates query and document into one sequence fed through a Transformer together, letting every token on both sides attend to each other inside the network (early interaction).
bi-encoder (embedding) cross-encoder (reranker)
┌────────┐ ┌────────┐ ┌───────────────────────┐
│ query │ │ doc │ │ [query] [SEP] [doc] │
└───┬────┘ └───┬────┘ └───────────┬───────────┘
▼ ▼ ▼
vector q vector d (cacheable) through Transformer together; tokens attend
└──cos(q,d)──┘ ▼
fast, offline, scans whole DB output one relevance score
but sides scored apart → precision high precision, but (q,d) bound → new query
ceiling voids all → not cacheable, N passes → rerank
top-K only
The mechanism reveals they're complementary: because the cross-encoder scores query and doc bound together, a new query voids everything precomputed — it can't cache or reuse, and ranking N docs means N full forward passes, so it can't scan the whole DB and is only fit as a reranker for a small candidate set. The bi-encoder's vectors are precomputable and compared in one shot — perfect for a whole-DB coarse filter. So standard RAG retrieval is the two in relay:
millions of docs ──[bi-encoder embedding + ANN]──▶ top-100 candidates
│
[cross-encoder reranker]──▶ top-5
│
handed to the LLM to answer
This also explains why the RAG assistant on your site is "embed the query first, then pull the nearest chunks from pgvector" — that step is the bi-encoder coarse filter. On the conflict side: because embedding "compresses to one vector up front," it inherently discards fine-grained interaction, so expecting reranker-level precision from embeddings alone is self-contradictory — not a tuning problem, but a ceiling fixed by the mechanism.
One modern closing part: Matryoshka Representation Learning (MRL) [10]. During training it requires that "the first 64 dims, first 256 dims, first 768 dims… each work as a good embedding on their own," nested like Russian dolls. The payoff is very engineering: the same vector can use only its first 64 dims for coarse filtering (fast, memory-light) and the full dims for fine scoring — no need to store several models. OpenAI's text-embedding-3 letting you specify dimensions, and truncatable dims across vendors, rely on this. As for "which model is most accurate on your task," the industry standard is MTEB (Massive Text Embedding Benchmark) [11], a leaderboard spanning retrieval / clustering / reranking and more — not a single similarity number; recent strong models like E5 [12] even add "prepend an instruction to the text (e.g. Represent this sentence for retrieval:)" to steer one model toward different tasks.
Closing the loop with one causal chain
Strung into one causal chain to help you recall it later:
Computers only know numbers, so text must become vectors → one-hot is a vector too, but orthogonal and meaning-free (empty geometry) → an embedding lookup is mechanically just one-hot × W, so "it's all matrices" is correct → but a randomly initialized W has no semantics either; what's special is a training objective chiseling W's numbers into "geometry = meaning" → word2vec uses the distributional hypothesis + negative sampling to turn co-occurrence into direction; BERT uses self-attention + MLM to deform vectors by context; SBERT uses contrastive InfoNCE to pry open the anisotropic cone into comparable sentence vectors → because meaning is encoded in direction, use cosine, and vectors can be precomputed → retrieval is fast and generalizes → and because compressing to one vector discards interaction, it can only be the whole-DB coarse filter (bi-encoder), needing a cross-encoder reranker for precision.
One line to answer your original question: what's special about an embedding isn't that "it's a matrix," but that "a training objective placed every row of that matrix where geometric distance equals semantic distance." The matrix is the stage; the loss is the choreographer.
Extension hooks (go wherever you need)
If you want to go one layer deeper, some natural next stops — pick what you need:
- The negative-sampling distribution: why sample by "word frequency to the 0.75 power," where 0.75 comes from, and what it balances.
- Pooling strategies (mean vs
[CLS]vs last-token) and why they matter so much when a decoder-only model is used as an embedding backbone (e.g. E5-Mistral). - The ANN index itself: how HNSW (Hierarchical Navigable Small World) approximates nearest neighbors without scanning the whole DB — the name encodes the principle and deserves its own breakdown.
- Domain fine-tuning: in your MES / part-number world, whether to contrastively fine-tune an embedding, or patch it with instruction prefixes + a reranker — the cost/benefit trade-off of the two routes.
Tell me which one to go deep on, and I'll walk you through it with the same bottom-up method.
References
[1] Mikolov, T. et al. (2013). Efficient Estimation of Word Representations in Vector Space (word2vec). — https://arxiv.org/abs/1301.3781
[2] Mikolov, T. et al. (2013). Distributed Representations of Words and Phrases and their Compositionality (negative sampling). — https://arxiv.org/abs/1310.4546
[3] Pennington, J., Socher, R., Manning, C. (2014). GloVe: Global Vectors for Word Representation. — https://nlp.stanford.edu/pubs/glove.pdf
[4] Vaswani, A. et al. (2017). Attention Is All You Need (Transformer / self-attention). — https://arxiv.org/abs/1706.03762
[5] Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. — https://arxiv.org/abs/1810.04805
[6] Reimers, N., Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. — https://arxiv.org/abs/1908.10084
[7] Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? (anisotropy / cone). — https://arxiv.org/abs/1909.00512
[8] van den Oord, A., Li, Y., Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding (InfoNCE). — https://arxiv.org/abs/1807.03748
[9] Gao, T., Yao, X., Chen, D. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. — https://arxiv.org/abs/2104.08821
[10] Kusupati, A. et al. (2022). Matryoshka Representation Learning. — https://arxiv.org/abs/2205.13147
[11] Muennighoff, N. et al. (2022). MTEB: Massive Text Embedding Benchmark. — https://arxiv.org/abs/2210.07316
[12] Wang, L. et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5). — https://arxiv.org/abs/2212.03533