Start with the problem: how does one token "see" the others
Dig to the foundation. After the embedding lookup, every token in the sentence has become a vector — but at this moment each vector is isolated. The vector for the character "行" is identical in「銀行」(bank) and「行走」(walking), because table lookup only sees the token itself. For a token's representation to change with context, there must be a mechanism that lets each token haul information over from other tokens and mix it into its own representation.
That mechanism is attention. It has to answer three sub-questions at once, and the three roles Q, K, V exist precisely one-per-sub-question:
Sub-question ① What am I looking for? → Query
Sub-question ② Who matches what I'm looking for? → Key
Sub-question ③ Once matched, what do I actually take? → Value
A familiar analogy: this is a database query. The Query is your search condition, the Key is each record's index field, the Value is the field content actually returned on a hit. The difference — you'll see it shortly — is that attention doesn't "exactly hit one record"; it "gives every record a match score and takes back a score-weighted blend of all the Values." This article walks your four questions in order: where Q/K/V come from → how similarity is computed (is it cosine?) → how it decides whose value to take → where soft prompts' trainable vectors come from and how they get inserted into K/V.
Where Q, K, V come from: the same vector times three different projection matrices
First, "where does QKV come from." Mechanically the answer is very simple: each token's hidden vector x is multiplied by three trained projection matrices W_Q, W_K, W_V, yielding that token's q, k, v[2]:
hidden vector x_i of token i (dimension d_model, e.g. 768)
│
├──× W_Q (d_model × d_k) ──▶ q_i "what I'm looking for"
├──× W_K (d_model × d_k) ──▶ k_i "what I can be found by"
└──× W_V (d_model × d_v) ──▶ v_i "what you actually take once you hit me"
All three come from the same x_i, just projected into three different role spaces
Here lies the question most often skipped yet most worth pressing: why three different matrices? Can't we just compute similarity between x and x directly?
No — and the reason is mechanistic. If you used x_i · x_j as the match score, the operation is symmetric — i's score for j equals j's score for i — and every token's dot product with itself is almost always the largest, so everyone attends mostly to themselves and no context ever mixes in. But relations in language are asymmetric: the pronoun "it" looks for its antecedent, while the antecedent doesn't necessarily look for "it"; an adjective looks for the noun it modifies, and the noun's needs differ. Separating W_Q and W_K gives the model two independent linear transforms — a "way of asking" and a "way of being found" — so that i asking about j and j asking about i can score completely differently. And W_V is yet another independent set because "the features by which you're matched" and "the information you hand over once matched" are also two different things — a token can be found by its syntactic features yet hand over semantic content.
The three-layer framework once more: W_Q/W_K/W_V are mechanism-layer parts (three ordinary linear layers); but the behaviors of "which kind of token should attend to which" are not human-written rules — they're carved into these three matrices by the objective layer (next-token cross-entropy). The attention mechanism only provides the pipeline of "information can be hauled by score"; what flows through the pipeline is decided by the loss.
How similarity is computed: dot product, not cosine — and the difference has substance
Now your second question: is the similarity between q and k cosine similarity? No. Attention uses the (scaled) dot product, and this differs from cosine in substance, not just terminology.
Write out the relationship between the two:
dot product: q · k = |q| × |k| × cos(θ) ← angle AND both lengths participate
cosine similarity: cos(θ) = (q · k) / (|q||k|) ← lengths divided out, angle only, range [-1, 1]
in other words: cosine = a forcibly normalized dot product
attention's dot product = unnormalized, unbounded
Why does this difference matter? Because vector length is a usable degree of freedom in the dot product. With cosine, no matter how important a key is, its score caps at 1; with the dot product, the model can learn a longer key vector for a token, letting it "shout louder" and get matched with high scores by all queries (and conversely learn short keys for unimportant tokens to keep them quiet). Length becomes a signal channel the model can exploit. This is also why you use cosine in embedding retrieval (fair comparison across documents requires dividing lengths out) but raw dot products inside attention (the model should be free to turn volumes up and down) — both are called "similarity," but the two settings have opposite design needs.
And what's the "scaled" part? The full formula[2]:
Attention(Q, K, V) = softmax( Q·K^T / √d_k ) · V
term by term:
Q·K^T : the matrix of dot products of every query with every key (n × n)
row i, column j = q_i · k_j = "token i's raw match score for token j"
/ √d_k : divide by the square root of the key dimension — the scaling
softmax : turn each row of scores into weights summing to 1 (next section)
· V : weighted average over all values with those weights
Why divide by √d_k? A detail asked a thousand times and worth settling. Suppose each component of q and k is independent with mean 0 and variance 1; then the dot product q·k = Σ q_m k_m is a sum of d_k random terms, so its variance is d_k — the higher the dimension, the larger the scores get, naturally. At d_k=64 scores easily hit ±20, ±30 — and what happens when such numbers enter softmax? After exponentiation the largest term monopolizes nearly all the weight, softmax saturates toward one-hot, gradients at all other positions vanish, and training stalls. Dividing by √d_k pulls the variance back to 1, keeping softmax in the regime where gradients exist. This isn't aesthetics — it's the engineering needed for deep networks to train at all.
How it decides whose value to take: not "selection" — a softmax-weighted blend of everything
Your third question — how does it decide which tokens' values to take — has an answer that may invert your intuition: it doesn't select. It takes every token's value, blended by softmax weights.
Softmax does two things here, step by step:
token i's raw scores against each token: s = [ 2.1, 0.3, -1.0, 4.0 ]
│
① exponentiate e^s: [ 8.2, 1.3, 0.37, 54.6 ] ← all positive, and gaps AMPLIFIED
│ (score gap 1.9 → ratio gap 6.7×)
② divide by the sum: [ 0.13, 0.02, 0.005, 0.85 ] ← a probability distribution, sums to 1
│
output_i = 0.13·v_1 + 0.02·v_2 + 0.005·v_3 + 0.85·v_4
= a convex combination (weighted average) of ALL values; the weights ARE "attention"
So "deciding whose information to take" is a soft decision: high-scoring tokens contribute most, low-scoring ones contribute nearly zero but never exactly zero. This isn't a compromise — it's deliberate: a hard selection (pick only top-1) makes "who was chosen" a discrete act with no gradient, and backpropagation can't pass through; softmax keeps the whole path differentiable, which is the only reason "who to attend to" can itself be learned by gradients. This also picks up the thread from the rerank article: the cross-encoder's early interaction is exactly this QK alignment running layer by layer between query tokens and document tokens.
One more mechanism must be added: the causal mask. In decoders like GPT, token i may not look at the future. The implementation is blunt and elegant: set the scores of all positions j > i to -∞ before softmax; after exponentiation they become 0, and not a drop of future value mixes in.
Finally, multi-head attention[2]: the whole procedure above runs not once but h times in parallel (e.g. 12 heads), each head with its own set of W_Q/W_K/W_V, projecting x into lower-dimensional (d_model/h) subspaces to do attention independently. Why? Because one softmax weighted-average can only express "one alignment pattern" — a head busy aligning syntactic dependencies cannot simultaneously use a different set of weights to align coreference. Multiple heads let different heads learn different relation types in different subspaces, then the h outputs are concatenated and fused through W_O. Post-hoc analyses indeed observe heads specializing: some fixate on adjacent positions, some track pronoun antecedents, some watch punctuation.
Soft prompts: where the trainable vectors come from, and how they're inserted into K/V
Now your fourth question — the moment we open the black box left in the PEFT article. Prefix-Tuning says "prepend trainable prefixes to K/V at every attention layer"[3] — inside that sentence are two concrete mechanical questions: what are these vectors, where do they come from? And what does "prepend" actually do, mathematically?
Where the trainable vectors come from. The answer is plain enough to surprise you: they are a directly declared parameter matrix — nn.Parameter in PyTorch, of shape (prefix length L_p) × (num_layers × 2 × d_model) (each layer needs one K-prefix and one V-prefix). They are not looked up from any token embedding and correspond to no real text; initialization is just random numbers (or seeded with embeddings of some real words, which converges more stably). Their "meaning" is entirely carved after the fact: during training the whole model is frozen and the loss's gradients flow only into this matrix, backpropagation step by step molding these numbers into "the shape that, when placed in KV, best pushes the model's behavior toward the task." It is pure gradient sculpture — nobody knows, or needs to know, what each dimension means.
How they're put in. Mechanically it's a matrix concatenation, and in implementation it enters through the existing past_key_values interface — the same doorway as the KV cache. The KV cache's original purpose: during autoregressive generation, the K/V of earlier tokens are computed once and stored, each new token computes only its own, and attention concatenates the cache back in. Prefix-Tuning borrows the same doorway to stuff a "fake history" at the very front:
attention at some layer, some head, with the prefix added:
K' = [ P_k ; k_1 k_2 ... k_n ] P_k: L_p × d_k ← trainable, directly a parameter
V' = [ P_v ; v_1 v_2 ... v_n ] P_v: L_p × d_v ← trainable, directly a parameter
↑ ↑
concatenated computed from real tokens via W_K/W_V (frozen throughout)
Note: Q has NO prefix! The prefix never asks — it is only "seen."
score_i = q_i · [P_k ; k_1..k_n]^T / √d_k ← every real token's query now has
L_p extra matchable targets
α_i = softmax(score_i) ← the distribution gains L_p positions
out_i = Σ α_ij·v_j + Σ α_im·P_v[m] ← the output now mixes in prefix values
Read the mathematical consequences and the soft prompt's working principle becomes fully transparent:
- The prefix contributes new keys → the softmax denominator gains L_p terms → every real token's attention distribution has some weight "siphoned off" to the prefix → the prefix then uses its own values to inject its trained bias signal into every token's output, in proportion to that weight. It amounts to a set of always-present, learnable "virtual memory entries" that keep seasoning the whole sentence at every layer.
- The prefix has no query of its own and never asks, so it reads nothing from the sequence — it only emits influence one-way. That is why so few parameters can shift model behavior so broadly.
A training-stability detail (reparameterization). Directly gradient-descending on P is unstable in practice (few parameters, high dimension, prone to oscillation), so Prefix-Tuning doesn't learn P directly during training but a much smaller matrix P′ passed through a small MLP: P = MLP(P′). After training, the MLP and P′ are discarded and only the final computed P is stored for deployment[3].
Contrast group to nail the levels. Same "soft prompt," but there are two places to put it, with different mechanistic consequences:
Prompt Tuning[4]: concat L_p trainable vectors ONLY in front of the input embedding
sequence; afterwards they travel the whole network like ordinary
tokens — each layer's K/V arise naturally from them flowing
through W_K/W_V.
(touches input, not middle layers; fewest params; matches full
fine-tuning only at large scale)
Prefix-Tuning[3]: skips the embedding layer and concatenates trainable matrices
directly into EVERY layer's K/V — independent virtual memory
per layer, much stronger control.
(P-Tuning v2[5] verified this "every layer" variant is robust
across scales)
LoRA[6] (contrast): touches neither sequence — it modifies the projection matrices
W_Q/W_V themselves (adding low-rank ΔW). One edits the "input
stream," the other edits the "parts" — this is the mechanistic
root of why soft prompts consume context and LoRA doesn't.
A closing table: today's machinery laid out straight
Object What it is How it comes about Trained?
─────────────────────────────────────────────────────────────────────────────────
x (hidden vec) token's current repr embedding lookup + prior layers —
W_Q/W_K/W_V three linear projections carved by pretraining gradients full FT: yes / PEFT: frozen
q, k, v x's three role projections x times the matrix, on the fly (follows above)
score q·k / √d_k dot product (NOT cosine — —
length carries signal)
attention wts post-softmax distribution exponentiate + normalize, sum=1 —
output Σ weight × value convex combination of all values —
P_k, P_v soft prompt's virtual nn.Parameter, random init, only this block
KV entries gradient-carved; concatenated
into every layer's K/V via
past_key_values
Closing: one logic chain tying the whole article together
After embedding lookup every token is isolated, so a mechanism is needed to haul information between them → attention splits the job into three roles — asking (Q), being found (K), handing over content (V) — all three obtained from the same x times three trained projection matrices → Q and K are separate so matching can be asymmetric (i seeking j ≠ j seeking i), and V is separate because "features you're matched by" ≠ "content you hand over" → similarity uses the dot product, not cosine, because vector length is a volume knob the model can exploit; dividing by √d_k tames the variance of high-dimensional dot products, preventing softmax saturation and vanishing gradients → "whose value to take" is no hard selection but a convex combination of all values under softmax weights — soft is differentiable, and only the differentiable can be learned; the causal mask zeroes the future with -∞ → multi-head lets different subspaces learn different alignment relations in parallel → a soft prompt then inserts virtual entries into this machinery: an nn.Parameter block, randomly born and gradient-carved, concatenated at the front of every layer's K/V via past_key_values, only seen and never asking, injecting learnable bias into every token's output through the weight softmax siphons off; training freezes the whole model, stabilizes via MLP reparameterization, and keeps only P at the end → Prompt Tuning edits the input layer, Prefix-Tuning edits every layer's KV, LoRA edits the projection matrices — three edits to three different parts of the same attention blueprint.
One line for the whole article: attention = a differentiable information-hauling pipeline of "q·k dot-product scoring (length-aware, divided by √d_k) → softmax weighting → value blending"; Q/K/V are three trained projections of the same vector, and a soft prompt is merely a block of "randomly born, gradient-carved" virtual memory concatenated into K/V, pouring task bias into every layer through softmax's weight channel.
Extension hooks
To go deeper, pick from: the KV cache's memory bill and how MQA/GQA (multiple query heads sharing K/V) cut it; how positional information enters QK (from additive positional encodings to RoPE rotations); the attention-sink phenomenon (why the first token often absorbs massive attention) and streaming inference; and how FlashAttention reorders computation to save memory without changing the math. Any one could be its own article.
References
[1] Bahdanau, D., Cho, K., Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate (the origin of attention). — https://arxiv.org/abs/1409.0473
[2] Vaswani, A. et al. (2017). Attention Is All You Need (Transformer / scaled dot-product / multi-head). — https://arxiv.org/abs/1706.03762
[3] Li, X. L., Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. — https://arxiv.org/abs/2101.00190
[4] Lester, B., Al-Rfou, R., Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. — https://arxiv.org/abs/2104.08691
[5] Liu, X. et al. (2022). P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. — https://arxiv.org/abs/2110.07602
[6] Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models (contrast: edits projection matrices, not the sequence). — https://arxiv.org/abs/2106.09685