Why LoRA Exists: What Actually Makes Full Fine-Tuning Expensive
To understand what each LoRA parameter really adjusts, start from the problem it was built to solve: the cost of full fine-tuning is not the compute — it is the storage and the optimizer state.
A 7B decoder-only model is 14GB of weights in bf16. But the real killer during training is the optimizer: AdamW keeps two extra fp32 states per trainable parameter (first-moment momentum and second-moment variance), which alone is 7B × 8 bytes ≈ 56GB; add fp32 master weights and gradients and you easily blow past 100GB. Worse, every task you fine-tune produces a full copy of the 7B weights — if you want separate models for "MES work-report extraction," "part-number description normalization," and "quality-anomaly classification," storage and deployment costs stack linearly.
LoRA (Low-Rank Adaptation) [1] makes one core bet: the change in weights during fine-tuning, ΔW, lives in a far lower-dimensional space than the weights themselves. The pretrained model already knows most of the language; fine-tuning is a small directional correction on top of existing capability — and there is no reason that correction needs all d×k degrees of freedom. So LoRA freezes W and learns only a ΔW that is structurally confined to a low-rank subspace.
The term "low-rank" deserves unpacking first: the rank of a matrix is the maximum number of linearly independent rows (or columns) — geometrically, "how many independent directions this linear map actually uses." A 4096×4096 matrix can have rank up to 4096; if its rank is only 8, it squeezes all information from the 4096-dimensional input space through an 8-dimensional channel before expanding again. LoRA imposes exactly that constraint, on purpose.
The Core Mechanism: the B·A Decomposition, and What r×k vs d×r Each Do
LoRA's modified forward pass is:
h = W₀·x + ΔW·x = W₀·x + (α/r) · B·A·x
W₀ ∈ R^(d×k) frozen pretrained weight (d = output dim, k = input dim)
A ∈ R^(r×k) trainable, the "compress / read" matrix
B ∈ R^(d×r) trainable, the "expand / write" matrix
r ≪ min(d,k) the rank, i.e. bottleneck width
α scaling constant (lora_alpha)
These are exactly the two matrices you asked about. Their dimension layout alone dictates their division of labor. Draw the data flow:
x (k-dim input)
│
├────────────────► W₀ (d×k, frozen) ────────► W₀x (d-dim)
│ │
└──► A (r×k) ──► Ax (r-dim bottleneck) ──► B (d×r) ──► BAx (d-dim)
▲ │
only r channels │ × (α/r)
▼
h = W₀x + (α/r)BAx
- A (r×k) adjusts "what to read from the input." It projects the k-dim input x down to r dims — each row of A is a k-dim vector that takes one inner product with the input, producing one scalar. So A defines r "detection directions": the input's components along those r directions get read out; information in the remaining k−r dimensions is discarded at this step. It is a lossy compression.
- B (d×r) adjusts "what to write to the output." It expands the r-dim bottleneck vector back to d dims — each column of B is a d-dim "write direction." The value in bottleneck channel i determines the coefficient with which B's i-th column gets added into the output.
- Together, ΔW = BA has rank at most r. That is a basic property of matrix multiplication: rank(BA) ≤ min(rank(B), rank(A)) ≤ r. No matter how training updates A and B, ΔW is forever trapped in the subspace of "at most r independent correction directions." LoRA's parameter efficiency is not a compression trick — it is a structural ceiling on expressiveness.
The parameter accounting is straightforward: full fine-tuning touches d×k parameters; LoRA touches r×(d+k). For a 4096×4096 attention projection with r=8:
full FT: 4096 × 4096 = 16,777,216 params
LoRA: 8 × (4096 + 4096) = 65,536 params (≈ 0.39%)
One more crucial inference-time property: after training you can compute W' = W₀ + (α/r)BA and merge ΔW back into the original weights — identical architecture and latency to the base model, zero extra cost [1]. This is LoRA's mechanistic advantage over Adapters (small modules inserted between layers, adding a serial computation at inference): an Adapter's extra module cannot be bypassed on the inference path, while LoRA's bypass can be mathematically absorbed.
What r Really Adjusts: the "Degrees-of-Freedom Budget" for the Correction
Hold on to the geometric picture above and r stops being abstract: r is the cap on the number of independent correction directions your fine-tune is allowed to apply to each weight matrix.
At r=1, ΔW = b·aᵀ is a rank-1 matrix: the entire correction has exactly one pattern — "when the input aligns with direction a, push the output along direction b," with only the strength varying per input. r=8 is a superposition of eight such detect→write pairs. The larger r is, the richer the transformations ΔW can express; in the limit r = min(d,k), LoRA matches full fine-tuning in expressiveness (but saves nothing).
So how large should r be in practice? It depends on the intrinsic dimensionality of the ΔW your task demands, and there is empirical evidence: Biderman et al. [6] found in code and math domains that the weight perturbations learned by full fine-tuning have rank 10–100× higher than typical LoRA configurations; in standard low-rank settings, LoRA clearly underperforms full FT at "learning new things" (especially continued pretraining that injects new knowledge) — but conversely, LoRA damages the base model far less: catastrophic forgetting is markedly lighter, better than regularizers like weight decay and dropout. In other words, r is simultaneously a capacity ceiling and a forgetting brake — two faces of the same mechanism (the low-rank constraint).
Translated into engineering judgment:
Task nature Rank ΔW needs Suggested start
────────────────────────────────────────────────────────────────────
Output format / style alignment low r = 8–16
(force JSON output, company
report tone, instruction tuning)
Moderate domain adaptation medium r = 16–32
(support-chat style + some
domain vocabulary)
New knowledge / new language / high r = 64–256 + rsLoRA
continued pretraining (or consider full FT)
In your setting: fine-tuning a model to extract MES work reports into a fixed JSON schema is behavior shaping — the model already reads Chinese and writes JSON; you are calibrating output habits, and r=8–16 is plenty. But making the model "internalize" the semantics of an entire part-numbering system (which field encodes package size, which series are siblings) is fact injection — a low-rank ΔW cannot hold it, and [6] suggests that even at high rank LoRA injects knowledge less efficiently than full FT. That need is usually better served by RAG: LoRA teaches behavior, retrieval supplies facts — mechanistically complementary, since behavior is a low-rank offset in the weights while facts are injected at inference time through the context window; they do not interfere.
One common trap: doubling r ≠ doubling quality. Once the task's intrinsic rank is satisfied, extra rank only buys overfitting room. The right procedure is to start small and increase r only when validation loss saturates — not to default to large.
The α/r Scaling: Why Divide by r, and How It Entangles with the Learning Rate
The (α/r) coefficient in the forward pass is the part most often memorized as a black box. Unpack it term by term:
α (lora_alpha) is a fixed constant, not trained; it sets the overall volume of the LoRA bypass relative to the trunk W₀x. Dividing by r exists so that you don't have to retune hyperparameters when sweeping r [1]: the original paper states that under Adam, tuning α is roughly equivalent to tuning the learning rate, so they simply set α to the first r they tried and never touched it again.
Why does dividing by r accomplish that? Intuition: BA·x is a sum of r rank-1 components —
BAx = Σᵢ bᵢ · (aᵢ·x) i = 1 … r
each term: reading of detection direction i × write vector i
More rank means more summed terms and a larger expected output scale. Dividing by r cancels that growth, keeping the bypass magnitude roughly comparable between r=8 and r=64, so your tuned learning rate survives a change of r.
But this design has a flaw identified later: 1/r over-corrects. Kalajdzievski [2] showed that a sum of r approximately independent random components grows like √r (variances add; standard deviation is the square root — the same reason a random walk ends up ~√r away after r steps), not like r. Under α/r scaling, the bypass's effective output therefore shrinks at rate 1/√r as r grows — which is why classic LoRA often "feels no different" going from r=8 to r=256: high rank isn't useless, the scaling factor is crushing the learning signal. The fix, rsLoRA (rank-stabilized LoRA), replaces the denominator with √r so the output scale stays O(1) at any rank, and the benefit of high rank actually materializes. In PEFT it is one switch: use_rslora=True, making the scale α/√r [9].
The practical rules that follow:
- Low rank (r ≤ 32): classic α/r is fine. Convention: α = r or α = 2r (α=2r pins the bypass gain at 2; widely used). Raising α ≈ raising the effective learning rate of the LoRA branch — don't sweep both at once.
- High rank (r ≥ 64): turn on rsLoRA, or you pay high-rank parameter cost without receiving high-rank capacity.
- When changing r, keep the α/r (or α/√r) ratio stable rather than moving r alone — otherwise you silently changed two hyperparameters and your ablation is contaminated.
Gaussian Random Initialization: Why "A Random, B Zero" and Not Anything Else
This is the part you asked to go deep on, and it is genuinely the most elegant step in LoRA's design. The original initialization: A gets a random Gaussian initialization, B is initialized to zero, so ΔW = BA is exactly zero at the start of training [1]. That single sentence hides three "why"s. Take them one at a time.
Why ΔW must start at zero
Because at t=0, h = W₀x + 0 = W₀x: the LoRA-equipped model is bit-for-bit identical to the base model. Fine-tuning starts from the pretrained model itself, not from a degraded version perturbed by random noise. If both A and B were random (init_lora_weights=False in PEFT — documented as debugging-only [9]), the first gradient step would face a model already pushed off the pretrained solution: you shove the model downhill first, then make it climb back.
Why you can't set both to zero — the gradients tell you
This is the key to the whole design. Write the scale as s = α/r and δ = ∂L/∂h (the loss gradient at this layer's output, d-dim). The gradients of the two LoRA matrices are:
∂L/∂B = s · δ · (Ax)ᵀ ← depends on A: the bottleneck activation Ax is its "input feature"
∂L/∂A = s · Bᵀ· δ · xᵀ ← depends on B: the gradient must pass *through* B to reach A
Suppose A = B = 0: then ∂L/∂B = δ·0ᵀ = 0 and ∂L/∂A = 0·δxᵀ = 0. Both gradients are identically zero — a saddle point that gradient descent can never leave. LoRA would learn nothing, forever. It is the same symmetry problem as why you can't initialize a neural network to all zeros: one side must break the symmetry.
The "A random, B zero" combination forms an elegant bootstrap sequence: at t=0, ∂L/∂A = Bᵀδxᵀ = 0 (A does not move on the first step!), but ∂L/∂B = δ·(Ax)ᵀ ≠ 0, because A is random and Ax is a set of nonzero random features. So B moves first: it learns to write "the features read out by A's r random projections" into the output in whatever way best reduces the loss. Once B is nonzero, gradient flows back through B into A, and A starts rotating its detection directions from "random" toward "useful for the task." The random A's role is to hand B a set of raw material it can use immediately.
Why the randomness is Gaussian, and how large the variance should be
Gaussian (normal-distribution) initialization delivers two mechanistic guarantees.
First, diversity across the r channels. Independent Gaussian random vectors in high dimension have a counterintuitive property: they are almost orthogonal (the cosine of the angle between two random directions concentrates near 0 with high probability, more tightly as dimension grows — the concentration-of-measure phenomenon). A's r rows are r independent k-dim Gaussian vectors, hence approximately mutually orthogonal — from step one, the r bottleneck channels each read a different linear combination of the input rather than collapsing into duplicate detectors. If the rows started out similar, their gradients would be similar too, they would stay redundant after training, the effective rank would be far below r, and you would have paid for parameters you never used.
Second, scale control of activations and gradients. This is exactly the problem He et al. (Kaiming initialization) [5] systematized: a signal passing through a linear layer has output variance equal to the sum of per-term contributions. If each component of x has variance 1 and A's entries are ~ N(0, σ²), each component of Ax has variance k·σ². Too large a σ² and the bottleneck activation explodes — and with it the first-step gradient ∂L/∂B = δ(Ax)ᵀ; too small and B receives a feeble signal and starts sluggishly. The remedy is to scale σ² inversely with fan-in (here, k): σ² ∝ 1/k, making the variance of Ax independent of k and O(1). [5]'s original contribution was deriving σ² = 2/fan_in for ReLU networks (the factor 2 compensating for ReLU zeroing half the signal); LoRA's A reuses the same variance engineering, just applied to the bypass.
One engineering-reality detail: HF PEFT's default (init_lora_weights=True) follows the Microsoft reference implementation and actually uses Kaiming-uniform for A (the uniform-distribution variant at the same scale) rather than a strict Gaussian; 'gaussian' is an explicit option (a normal distribution scaled relative to the rank); in both cases B is always zeroed and variance is fan-in scaled [9]. The shape difference between uniform and Gaussian is practically imperceptible — what is actually load-bearing is the trio: zero-product start + variance scale + channel diversity.
Why "B zero, A random" and not the mirror image
On the surface the two are symmetric — both give ΔW = 0. But Hayou et al. [4] showed the two schemes produce completely different training dynamics: the B=0, A-random scheme (they call it Init[A]) tolerates larger learning rates without output instability and performs better on average. Intuitively, A is wired to the k-dim input and B to the d-dim output; the two sides have asymmetric scale sensitivities, and letting the "read side" carry the random structure while the "write side" starts cautiously from zero gives a gentler perturbation path into the output. This echoes the same authors' finding in LoRA+ [3]: A and B fundamentally should not share one learning rate — LoRA+ assigns B a much larger learning rate than A (at a fixed ratio), yielding 1–2% quality gains and up to ~2× faster convergence on wide models. A and B are mechanistically different objects; both their initialization and their learning rates deserve different treatment.
The ceiling of random init, and the informed alternatives
A random A provides "usable raw material," not "a good starting point" — the initial directions are task-agnostic, so early convergence is slow. This spawned a whole family of smarter initializations, mechanically diverse but sharing one idea: start from informative directions, not from noise:
PiSSA [8] SVD of W₀; initialize A, B with the principal singular
values/vectors so fine-tuning acts directly on the weight's
most important subspace; the residual is frozen
LoRA-GA estimate the first few full-FT gradient steps, SVD them,
and start LoRA along the direction full FT would have taken
EVA SVD of layer input activations (data-driven), allocating
per-layer rank by explained variance
LoftQ for quantized backbones: initialize A, B to compensate
W₀'s quantization error (the QLoRA setting)
All of these are just string options of init_lora_weights in PEFT [9]. Note the level distinction: they only change the starting point — the low-rank bottleneck, the B·A decomposition, and the α scaling all remain untouched.
A Practical Tuning Table
Condensing all the mechanics into one decision table (each row derived from mechanism, not memorized):
Knob What it adjusts (mechanism) Practical guidance
──────────────────────────────────────────────────────────────────
r cap on ΔW's independent format/style: 8–16
correction directions = domain adapt: 16–32
capacity vs forgetting lever knowledge: 64+ with rsLoRA,
or full FT / RAG instead
──────────────────────────────────────────────────────────────────
α overall bypass gain; acts as start at α = r or 2r;
the LoRA branch's effective if you tune α, don't
learning rate [1] sweep lr simultaneously
──────────────────────────────────────────────────────────────────
scale denom α/r: signal shrinks at high r ≥ 64 → use_rslora=True
rank [2]; α/√r: scale is (α/√r)
rank-independent
──────────────────────────────────────────────────────────────────
init A Gaussian/Kaiming, B zero = default is fine; for faster
zero start + symmetry convergence try pissa /
breaking [1][4] eva [8][9]
──────────────────────────────────────────────────────────────────
target which matrices get a bypass original paper: q,v only [1];
modules modern practice leans
all-linear (incl. MLP) [6]
──────────────────────────────────────────────────────────────────
learning LoRA params start from typically ~1e-4 (about an
rate random/zero, not from a order above full FT);
pretrained point advanced: LoRA+ gives B a
larger lr [3]
──────────────────────────────────────────────────────────────────
low memory quantized trunk + LoRA QLoRA: frozen 4-bit backbone,
65B fine-tuned on one
48GB GPU [7]
Two common debugging patterns also fall straight out of the mechanics: "increasing r does nothing" — first check whether α/r scaling is crushing the signal (switch to rsLoRA), then whether the task is intrinsically low-rank (r already saturated); "severe capability regression after fine-tuning" — r or lr too large, you dismantled the low-rank brake yourself; lower r or restrict to attention layers.
Closing: One Logical Chain
Stringing the whole article together: full FT is expensive because every task duplicates the full weights plus optimizer state → but fine-tuning's ΔW is intrinsically low-dimensional and doesn't need d×k degrees of freedom → so B(d×r)·A(r×k) confines ΔW to an r-dim subspace: A picks which r directions to read from the input, B picks which r directions to write to the output, and r is the correction's degrees-of-freedom budget → summing r components grows with r, so divide by α/r to stabilize scale and let α act as an equivalent learning rate; but 1/r over-corrects, and high rank needs 1/√r (rsLoRA) → for training to bootstrap, ΔW must start at zero without sitting on a saddle point, so B=0 guarantees the zero start and Gaussian-random A breaks the symmetry, with fan-in-scaled variance controlling signal scale and high-dimensional near-orthogonality keeping the r channels from collapsing → B moves first, A follows; the A/B asymmetry further implies they deserve different learning rates (LoRA+), and the random start can be replaced by an informed one (PiSSA) → finally, low rank is at once a capacity ceiling and a forgetting brake: teach behavior with LoRA, supply facts with RAG.
Natural next stops if you want to go deeper: DoRA (further decomposing ΔW into magnitude × direction), combining LoRA with MoE, multi-LoRA serving (one base model hot-swapping many adapters at inference), and the "intruder dimensions" geometry behind [6] (why LoRA's learned singular vectors end up nearly orthogonal to the pretrained weight's singular vectors).
References
[1] Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. — https://arxiv.org/abs/2106.09685 [2] Kalajdzievski, D. (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. — https://arxiv.org/abs/2312.03732 [3] Hayou, S., Ghosh, N., Yu, B. (2024). LoRA+: Efficient Low Rank Adaptation of Large Models. — https://arxiv.org/abs/2402.12354 [4] Hayou, S., Ghosh, N., Yu, B. (2024). The Impact of Initialization on LoRA Finetuning Dynamics. — https://arxiv.org/abs/2406.08447 [5] He, K., Zhang, X., Ren, S., Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. — https://arxiv.org/abs/1502.01852 [6] Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. — https://arxiv.org/abs/2405.09673 [7] Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. — https://arxiv.org/abs/2305.14314 [8] Meng, F., Wang, Z., Zhang, M. (2024). PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. — https://arxiv.org/abs/2404.02948 [9] Hugging Face PEFT Documentation — LoRA (LoraConfig: init_lora_weights, use_rslora). — https://huggingface.co/docs/peft/package_reference/lora