Following on: the vectors are made — now what?
The last two posts took embeddings apart: a training objective carves text into vectors so that "similar meaning" becomes "close geometry." But that's only the first half. The real retrieval scenario is this — your knowledge base holds a million, even a billion vectors (768 or 1024 dims each), a user throws in one query vector, and you must return the closest top-K in milliseconds.
That is the problem a vector database exists to solve. It's not shallowly "a place to store vectors" — you can store vectors in a single Postgres table. Its entire value is one thing: finding nearest neighbors in high-dimensional space, fast and accurately. This post goes from "why brute force won't do" all the way down to every gear inside HNSW, the operating logic of each vector DB, and exactly which knobs you can turn.
The root problem: why brute-force NN fails, and why we go "approximate"
The most intuitive approach is brute-force (flat) search: compute the distance from the query to every vector in the base, sort, take top-K. The problem is cost:
N vectors, d dims each, cost per query ≈ O(N × d)
N = 1,000,000 vectors × d = 768 dims
→ ~768 million multiply-adds per query, plus sorting 1M distances
→ a single machine handles only a handful of queries/sec (dismal QPS)
"So use a tree index?" In low dimensions (2D, 3D) kd-trees and R-trees cut search to logarithmic. But at hundreds or thousands of dimensions they all break — the curse of dimensionality: as dimension grows, all points drift to nearly-equal distances, space becomes "almost all boundary," the tree's pruning conditions almost never hold, and it degrades into a near-full scan — slower than brute force (with tree overhead on top).
So the industry changed the goal: stop insisting on "guaranteed exact nearest," and instead trade a tiny bit of accuracy for a huge speedup — this is ANN (Approximate Nearest Neighbor). Its quality metric is recall@K: of the top-K that ANN returns, how many are truly in the exact brute-force top-K.
recall@10 = 0.95 means: of the 10 results ANN returns, on average 9.5
match the exact top-10 that brute force would compute
the whole game of a vector DB = find the sweet spot between
recall (accurate) and latency/QPS (fast)
Hold onto this recall↔latency trade-off axis — every parameter later is, at bottom, sliding along it.
Enter the protagonist: HNSW — unpack the name and you have the principle
The default ANN index in today's mainstream vector DBs (Qdrant, Weaviate, Milvus, pgvector, Pinecone…) is almost always HNSW [1]. Its name — Hierarchical Navigable Small World — encodes the principle; let's unpack it word by word.
Small World: from "six degrees of separation" — if every node in a graph connects not only to its nearby neighbors but also to a few long-range shortcuts, then the shortest path between any two nodes shrinks to O(log N) hops. This is the geometric basis of HNSW's speed.
Navigable + why long-range links matter: a neighbor graph alone isn't enough — you must be able to "walk" it. HNSW uses greedy search: start from an entry point, look at the current node's neighbors, jump to the one closest to the query, repeat, until no neighbor is closer than the current node. The catch — if the graph has only short-range links, greedy easily gets stuck in a local optimum (nothing nearby is closer, but something globally is, and you can't hop to it). This was exactly the insight of the earlier NSW [2] papers: add long-range links so greedy can first stride toward the target region, then refine with short links.
Hierarchical: HNSW adds one more trick on top of NSW — split the graph into multiple layers, like a skip list. The highest layer a node is assigned to is chosen randomly with an exponentially decaying probability: the vast majority of nodes live only on the bottom layer, a few reach higher layers.
Layer 2 ●─────────────────────● sparse, long hops: cross big ground fast
\ /
Layer 1 ● ●────────●────────● ● medium density
\ \ | / /
Layer 0 ●●●●●●●●●●●●●●●●●●●●●●●●●●● bottom layer: all nodes, densest, fine search
▲
query enters at the top-layer entry point, greedily walks to that
layer's local nearest, descends a layer, repeats, and at Layer 0
uses a candidate list to pick the final top-K
Search flow: start greedy from the single top-layer entry point, reach this layer's nearest → use it as the start, descend a layer, greedy again → … → at Layer 0, maintain a dynamic candidate list of size ef (the nearest ef found so far), and take top-K from it. Upper layers "stride into the right neighborhood," lower layers "refine" — together giving O(log N) query complexity [1].
Building is the reverse — an insertion process: a new node arrives, its top layer is picked randomly, then from the top it greedily finds neighbor candidates on each layer (the candidate width is efConstruction), selects the best M links per layer, and builds bidirectional edges.
HNSW's tunable parameters (what each knob does mechanically)
Since you'll actually tune these, here are the three core parameters derived from the mechanism — all live on the recall↔latency/memory axis:
┌─────────────────┬──────────────────────────────┬───────────────────────────────┐
│ Parameter │ what it is mechanically │ effect of raising it │
├─────────────────┼──────────────────────────────┼───────────────────────────────┤
│ M │ links per node per layer │ recall↑, memory↑, build slower│
│ (build time) │ (graph density) │ typical 16~64 │
│ efConstruction │ candidate width per node │ graph quality↑ (recall↑), │
│ (build time) │ while building │ build slower; typical 100~200 │
│ efSearch (ef) │ dynamic candidate-list size │ recall↑, latency↑ │
│ (query time) │ at query │ the only knob tunable LIVE │
└─────────────────┴──────────────────────────────┴───────────────────────────────┘
Key distinction: M and efConstruction are baked into the graph structure at build time — changing them means rebuilding the index; efSearch takes effect at query time — so to switch between "this batch needs to be more accurate" and "faster" online, the knob you turn is efSearch. That's the bottom-level reason pgvector's SET hnsw.ef_search = 100; applies instantly, while m / ef_construction must be fixed at CREATE INDEX.
The other road: IVF and PQ (a contrast set that reveals HNSW's trade-offs)
HNSW isn't the only answer. Understanding two other families shows where HNSW's trade-offs lie.
IVF (Inverted File): first use k-means to cluster all vectors into nlist groups (each a centroid, forming a Voronoi partition). At query time, find which centroids the query is closest to, and do exact scoring only inside the nearest nprobe groups, skipping the rest.
carve space into nlist Voronoi cells:
┌───────┬───────┬───────┐
│ c1 │ c2 │ c3 │ query ★ falls near c5
├───────┼───★───┼───────┤ → scan only the nearest nprobe cells (e.g. c5,c2,c6)
│ c4 │ c5 │ c6 │ → ignore all other cells
└───────┴───────┴───────┘
larger nprobe → more cells scanned → recall↑, slower (same trade-off axis again)
IVF's parameters are nlist (how finely to split) and nprobe (how many cells to probe). Versus HNSW, IVF builds faster and uses less memory, but is usually slower at the same recall — good for very large, mostly-static datasets.
PQ (Product Quantization) [3]: this is a compression technique for "too many vectors to fit in RAM." It splits each d-dim vector into m sub-vectors, quantizes each into an 8-bit code via a small codebook (learned with k-means, typically 256 representatives). A vector that was 768×4 bytes (3072 bytes) becomes m bytes (e.g. m=96 → 96 bytes), and distances are approximated via table lookups + sums. The cost is precision loss. Often combined with IVF into IVFPQ (FAISS's signature combo [4]). Parameters: m (number of segments) and nbits (bits per segment).
Two others worth naming: ScaNN [5] (Google, anisotropic quantization, especially strong for MIPS inner-product search) and DiskANN [6] (Microsoft, based on the Vamana graph, puts the index on SSD so one machine handles billions). All of them pick different points on the same recall/speed/memory triangle.
How embeddings work with the vector DB (the full pipeline)
Back to your core question — how embeddings plug into the DB. The whole RAG retrieval pipeline:
[Ingest]
raw text (work orders/part #s/specs) ──embedding model──▶ vector ──(L2 normalize)──▶ insert into ANN index
+ store payload (text, line, category…)
[Query]
user question ──same embedding model──▶ query vector ──ANN search──▶ top-K ids
│
fetch payload by id ◀───────┘
│
(optional) cross-encoder rerank ──▶ hand to LLM
Three bottom-level details that must line up:
First, the same embedding model. Ingest and query must use the same model and same normalization — otherwise the two sides land in different spaces and geometric distance is meaningless.
Second, the distance metric must match how the model was trained. This pays off the previous post's thread: if the model was trained with cosine, the index must use cosine (pgvector's <=>); inner product → inner product (<#>); Euclidean → L2 (<->). Pick the wrong metric and recall goes mysteriously bad.
Third, metadata filtering (e.g. only part numbers on "line A, resistor category"). Here lurks an often-missed mechanism trap — pre-filter vs post-filter:
- post-filter (ANN first, then filter): take top-K vectors, then drop those not matching. Problem: if the condition is rare, top-K may leave only two or three after filtering — recall collapses.
- pre-filter (filter first, then ANN): shrink the candidate set by metadata first. Problem: HNSW is one global graph; block most nodes and greedy can't traverse — it can't hop where it should, so recall suffers too.
How each DB handles this is precisely where they pull apart in the next section.
A tour of the vector DBs (operating logic + tunables)
Most are HNSW under the hood, but their packaging, filtering strategy, and deployment form differ a lot. One by one:
┌────────────┬─────────────────┬───────────────────────┬────────────────────────────┐
│ Vector DB │ form │ underlying index │ traits / tunables │
├────────────┼─────────────────┼───────────────────────┼────────────────────────────┤
│ pgvector │ Postgres ext. │ hnsw / ivfflat │ lives in PG; m,ef_construc │
│ │ │ │ tion,ef_search; <=> <-> <#>│
│ FAISS │ library (not svc)│ Flat/IVF/HNSW/PQ box │ most flexible; wrap it your │
│ │ │ │ self; GPU support │
│ Milvus │ distributed svc │ HNSW/IVF/SCANN/DiskANN │ billion-scale, segments, │
│ │ │ │ many index types │
│ Qdrant │ service (Rust) │ HNSW │ strongest payload filtering,│
│ │ │ │ low latency │
│ Weaviate │ service │ HNSW │ built-in hybrid, embed mods │
│ Pinecone │ managed serverless│ proprietary (HNSW-ish)│ zero-ops, autoscale, rerank│
│ Chroma │ embedded │ HNSW │ easiest for dev/prototyping │
└────────────┴─────────────────┴───────────────────────┴────────────────────────────┘
Per-DB operating logic:
pgvector — what you're using. It's not a standalone DB; it grows vector search into PostgreSQL: the vector is a vector column, the index is CREATE INDEX ... USING hnsw, filtering is plain SQL WHERE, and you can JOIN with your existing relational data. Its biggest edge is "one DB for content + vectors + metadata," no extra system to run — exactly how your site's doc_chunks works.
FAISS — strictly a library, not a database: no service layer, no persistence or filtering out of the box, but it's the "toolbox ancestor" of all index algorithms (Flat/IVF/HNSW/PQ/IVFPQ are all in it), used or echoed under the hood by many vector DBs, with GPU support.
Milvus — built for billion-scale distributed search: data split into segments, compute/storage separated, the widest index support (including DiskANN and GPU indexes). Powerful but heavier to operate.
Qdrant — written in Rust, HNSW at its core, its standout is deep payload filtering: it folds filter conditions into the graph search (filterable HNSW), easing the "pre-filter can't traverse" problem, keeping low latency under complex filters.
Weaviate — HNSW-based, with built-in hybrid search (vector + BM25 keyword fusion — dovetailing with post #2's sparse/dense hybrid), embedding modules, and multi-tenancy.
Pinecone — fully managed serverless; you never touch index internals. It sells zero-ops, autoscaling to billions, built-in rerank. The cost is opacity and vendor lock-in.
Chroma — lightweight embedded, runs inside your Python process, easiest for dev and prototyping, not for large-scale production.
Landing it in your pgvector RAG
Concretely for your MES / part-number retrieval, pgvector looks like this:
-- build an HNSW index (cosine, to match a cosine-trained embedding)
CREATE INDEX ON doc_chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
-- tune recall/speed live before querying (no rebuild needed)
SET hnsw.ef_search = 100;
-- vector search with metadata filtering (pre-filter)
SELECT id, content
FROM doc_chunks
WHERE line = 'A' AND category = 'resistor' -- metadata condition
ORDER BY embedding <=> $1 -- <=> is cosine distance
LIMIT 10;
Practical recommendation, chaining the last two posts' conclusions: embedding (use BGE-M3's dense+sparse hybrid) → pgvector HNSW coarse filter top-100 → cross-encoder reranker top-5 → hand to the LLM. For part numbers where "one character off means a different part," lean on the sparse path of the hybrid; filter line/category with WHERE. Raise ef_search from 40 (default) toward 100~200, watch the recall-vs-latency curve, and find your sweet spot.
Closing the loop with one causal chain
embeddings produce vectors, but brute-force NN over millions is too slow and high dimensions break tree indexes (curse of dimensionality) → so we go ANN, trading a little recall for speed → HNSW uses small-world short+long links so greedy search arrives in a few hops, plus a hierarchy (skip-list style) that presses complexity to O(log N) → the knobs are M / efConstruction (build time, baked into structure) and efSearch (query time, live) — all sliding on the recall↔latency axis → IVF/PQ are another set of trade-offs (clustering saves compute, quantization saves memory) → embeddings plug into the index via "same model, same metric, same normalization," with metadata pre/post-filter → the DBs are mostly HNSW; they differ in filtering strategy and deployment; your pgvector wins by holding content + vectors + SQL filtering in one DB.
In one line: a vector DB's value isn't in "storing" — it's in "using an ANN index like HNSW to find you the tunable sweet spot between recall and latency."
Extension hooks
- HNSW's memory bill: how M directly sets each vector's graph overhead (~M×2 neighbor ids), and estimating RAM at billion scale.
- Quantization for memory in practice: pgvector's
halfvec(half precision) and scalar quantization — when it's worth it and how much recall you lose. - How filterable HNSW works inside: how Qdrant/Weaviate keep filtering from breaking graph connectivity.
- The build-cost curve: measured effect of efConstruction 64→200 on build time and recall, and when it stops paying off.
Tell me which to go deep on.
References
[1] Malkov, Yu. A., Yashunin, D. A. (2016/2018). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (HNSW). — https://arxiv.org/abs/1603.09320
[2] Malkov, Yu. et al. (2014). Approximate nearest neighbor algorithm based on navigable small world graphs (NSW). Information Systems. — https://doi.org/10.1016/j.is.2013.10.006
[3] Jégou, H., Douze, M., Schmid, C. (2011). Product Quantization for Nearest Neighbor Search. IEEE TPAMI. — https://doi.org/10.1109/TPAMI.2010.57
[4] Johnson, J., Douze, M., Jégou, H. (2017). Billion-scale similarity search with GPUs (FAISS). — https://arxiv.org/abs/1702.08734
[5] Guo, R. et al. (2020). Accelerating Large-Scale Inference with Anisotropic Vector Quantization (ScaNN). — https://arxiv.org/abs/1908.10396
[6] Subramanya, S. J. et al. (2019). DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node. NeurIPS. — https://papers.nips.cc/paper/2019/hash/09853c7fb1d3f8ee67a61b6bf4a7f8e6-Abstract.html
[7] pgvector — open-source vector similarity search for Postgres (docs). — https://github.com/pgvector/pgvector
[8] Qdrant documentation — filtering & HNSW. — https://qdrant.tech/documentation/
[9] Milvus documentation — index types. — https://milvus.io/docs
[10] Weaviate documentation — vector index & hybrid search. — https://weaviate.io/developers/weaviate
[11] Pinecone — Hierarchical Navigable Small Worlds (HNSW) explainer. — https://www.pinecone.io/learn/series/faiss/hnsw/
[12] Indyk, P., Motwani, R. (1998). Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality (LSH). STOC. — https://doi.org/10.1145/276698.276876