Start from the Real Problem: Why a 7B Model Won't Run on Your Machine
Push the problem to its root. A 7B (7-billion-parameter) model stored in FP16 (16-bit floating point, 2 bytes per parameter) needs 7B × 2 bytes = 14 GB for the weights alone. If the GPU in your office workstation has 8 GB of VRAM, the weights simply don't fit — that's the first wall: capacity.
But even with a 24 GB card there is a second, sneakier wall. An LLM generates text one token at a time, and for every single token, the GPU must stream the entire set of weights from memory into the compute units — every layer's matrix multiplication touches all of that layer's weights. In this phase (decode), the arithmetic itself is cheap (you're only processing one token's vector); the real bottleneck is how fast data can be moved, i.e. memory bandwidth. Engineers call this regime memory-bound: the ALUs spend most of their time waiting for data, not computing.
That gives generation speed a brutally simple ceiling:
tokens/s ceiling ≈ memory bandwidth / total weight bytes
GPU with 1000 GB/s running a 14 GB FP16 7B model
→ ceiling ≈ 1000/14 ≈ 71 tokens/s
Typical PC, dual-channel DDR5 ≈ 80 GB/s, same model on CPU
→ ceiling ≈ 80/14 ≈ 5.7 tokens/s (reality is lower)
Once you see this equation, you see why quantization kills two birds with one stone: compress each weight from 16 bits to 4 bits and 14 GB becomes 3.5 GB — the capacity wall falls, and the bytes moved per token drop to 1/4, pulling the theoretical generation speed up by ~4×. That's why the ecosystem is flooded with INT8, INT4, Q4_K_M, NF4 and FP8 models: these aren't marketing names for "crippled versions," they are an entire engineering discipline's systematic answer to those two walls.
We'll unpack it in this order: what a number physically looks like in memory (skip this and everything after is a black box) → the core math of quantization → why LLMs are unusually hard to quantize (the outlier problem) → the four main engineering routes around it (LLM.int8(), GPTQ, AWQ/SmoothQuant, NF4) → deployment formats like GGUF → what actually happens inside the kernel → and finally cost and selection.
The Bottom of the Bottom: What a Weight Looks Like in Memory
"Quantization" literally means "mapping continuous values onto a discrete grid." To understand it you first need to know how the original floating-point number represents a value — otherwise "compress FP16 to INT4" is just an incantation.
A float is binary scientific notation: (-1)^sign × 1.mantissa × 2^(exponent - bias). Each field has one job:
FP32 |s| 8-bit exponent | 23-bit mantissa | 4 bytes
FP16 |s| 5-bit exp | 10-bit mantissa | 2 bytes
BF16 |s| 8-bit exponent | 7-bit mantissa | 2 bytes
FP8
E4M3 |s| 4-bit exp | 3-bit m | 1 byte
E5M2 |s| 5-bit exp | 2-bit m| 1 byte
s = sign bit
exponent → sets DYNAMIC RANGE: how large/small a value can be (where the window sits)
mantissa → sets PRECISION: how finely you can resolve within that window
This division of labor between exponent and mantissa is the key to every low-precision format. BF16 and FP16 both take 2 bytes, but BF16 spends its bits on the exponent (8 bits — the same dynamic range as FP32) and sacrifices mantissa precision. That's why training prefers BF16 (gradient magnitudes swing wildly; overflow is the danger), while FP16 is finer-grained but narrow. FP8 pushes the same trade-off one notch further: E4M3 (better precision) for weights and activations, E5M2 (wider range) for gradients — a format jointly defined by NVIDIA/Arm/Intel, with native hardware support from Hopper onward [6]. Strictly speaking FP8 is "low-precision floating point" rather than classic quantization, but it attacks the same problem: fewer bytes per number.
Integer formats are a different animal: INT8 is just 256 evenly spaced grid points in [-128, 127]; INT4 has only 16 points in [-8, 7]. No exponent field means no automatic scaling — the grid spacing is fixed. Which raises quantization's central question: neural-network weights follow a bell-shaped distribution centered at 0 (roughly normal), with most values crowded in a small interval near zero. How do you map that continuous distribution onto 16 equally spaced points with minimal loss?
The Core Math: Squeezing a Bell Curve into 16 Buckets
Every uniform quantization scheme is the same affine map [7][8]:
quantize: q = clamp( round( x / s ) + z , q_min , q_max )
dequantize: x̂ = s · ( q − z )
x = original float value q = integer code
s = scale: how much real-value width one grid step represents
z = zero-point: which integer code real 0.0 maps to
round = snap to nearest grid point clamp = saturate out-of-range values
Term by term: x / s compresses the real axis into grid coordinates; round is the moment information is actually destroyed — every real value landing in the same bucket becomes indistinguishable; z lets asymmetric distributions (e.g. post-ReLU activations, all positive) use the full integer range. Weight distributions are roughly symmetric, so weights usually use symmetric quantization (z = 0), saving the storage and arithmetic of a zero-point.
How big is the error? Rounding costs at most half a step: |x − x̂| ≤ s/2. So the whole game has a single objective: make s as small as possible. And s is dictated by the range:
s = ( max(x) − min(x) ) / (number of grid points − 1)
This equation exposes the fatal weakness: s is held hostage by the single largest value in the distribution. One outlier stretches the range, s blows up, and the 99.9% of well-behaved values near zero get squeezed into a handful of buckets — like sizing the whole class's uniforms off the tallest kid.
The first-line engineering countermeasure is to shrink the region that shares one s, called quantization granularity:
per-tensor: one s for the whole matrix ← cheapest; one outlier ruins everything
per-channel: one s per row/column ← much better when rows differ
per-group: one s per 64/128 weights ← the standard for modern LLM quantization
Finer granularity → smaller per-group max → smaller s → smaller error; the cost is storing many more scales. Do the arithmetic: 4-bit quantization with one FP16 scale per 128 weights averages 4 + 16/128 = 4.125 bits per weight — a 3% space surcharge buying a large accuracy gain, which is why virtually every quantized model you download is group-wise.
That's the foundation shared by all methods. The next question: where does this foundation crack when you build an LLM on it?
Why LLMs Are Especially Hard: the Tyranny of Outliers
Apply the naive recipe above — calibrate, compute per-group scales, round to nearest (RTN) — and small models survive, but beyond ~6.7B parameters quality suddenly collapses. The LLM.int8() paper identified the culprit [1]: as transformers scale up, their activations (the intermediate outputs of each layer) systematically grow outlier features — concentrated in a few fixed hidden dimensions, roughly 20× larger than everything else, appearing in nearly every layer and nearly every token.
Note a crucial level distinction that trips many people up:
weights fixed after training, well-behaved distribution
(bell-shaped, symmetric, few extremes)
→ can be quantized offline, at leisure; easy
activations change with every input, must be quantized online,
and carry the outliers
→ THIS is the hard part, not the weights
What happens when an outlier meets "s held hostage by the max" is obvious once drawn:
one activation channel of some layer (illustrative):
values: 0.3 -0.5 0.8 -0.2 0.4 57.1 0.6 -0.7
↑ outlier (a fixed dimension)
scale from max=57.1 (INT8, symmetric): s = 57.1/127 ≈ 0.45
→ 0.3 quantizes to round(0.3/0.45) = 1
→ 0.4 is also 1, -0.5 is -1, 0.6 is 1 ...
→ all normal values collapse into {-1, 0, 1} — their differences erased
→ yet these "normal" dimensions carry most of the semantics
Worse, you can't simply clip the outliers away — experiments show these dimensions are disproportionately important to the model's output; clip them and perplexity (the standard measure of a language model's predictive quality — lower is better) explodes [1]. Outliers can't be removed yet ruin quantization: this dilemma is the actual problem every post-2022 LLM quantization paper is solving. The four mainstream routes are four different ways around it.
Route 1: Isolate the Outliers — LLM.int8()'s Mixed-Precision Decomposition
LLM.int8() takes the most direct approach: since outliers concentrate in a few fixed dimensions (~0.1% in practice), split the matrix multiplication in two — the outlier columns stay in FP16, the other 99.9% run in INT8, and the two partial results are summed [1]. Combined with vector-wise scaling (one scale per row/column of the matmul — finer than per-tensor), it achieves essentially lossless 8-bit inference on 175B-class models.
Derive its pros and cons from the mechanism: the upside is no retraining and no calibration data — it works at load time (this is what Hugging Face's load_in_8bit=True runs underneath). The downside also follows directly: every forward pass must detect outliers online, split the matrices, run two precision paths, and merge — overhead that makes it not necessarily faster than FP16 at small batch sizes. Its win is memory (14 GB → 7 GB), not latency. Which teaches a general lesson: quantization's benefits come in two separate ledgers — memory saved and speed gained — and one does not automatically imply the other.
Route 2: Compensate While Quantizing — GPTQ's Second-Order Math
RTN rounds each weight independently, which wastes a degree of freedom: weights are not isolated — an entire row of weights takes an inner product with the same input vector. The rounding error of one weight can be absorbed by slightly adjusting the not-yet-quantized weights, so that the final output error is minimized. That is the heart of GPTQ [2].
It swaps the objective from "minimize weight error" to "minimize output error," solved layer by layer:
argmin_Ŵ ‖ W·X − Ŵ·X ‖²
W = the layer's original weights Ŵ = quantized weights
X = this layer's inputs when a calibration set
(a few hundred real text passages) flows through
What's the difference? Weight error treats every weight as equally important; output error automatically discovers that "this weight multiplies inputs that are usually large — a small error here moves the output a lot." Importance is determined by input statistics (X·Xᵀ — mathematically, the Hessian of this objective, its matrix of second derivatives). GPTQ's algorithm descends from the 1990s Optimal Brain Surgeon: quantize one column at a time, then use the inverse Hessian to compute how the remaining columns should shift to absorb the error just created, and sweep to the end. With numerical engineering on top (lazy batch updates, Cholesky decomposition), a 175B model quantizes to 4-bit in about 4 GPU-hours [2].
Again derive the trade-offs from the mechanism: because errors are actively compensated, GPTQ beats RTN clearly at 4-bit and even 3-bit; the price is needing calibration data and an offline quantization pass, and the greedy layer-wise optimization risks overfitting to the calibration set — biased data yields a biased quantized model.
Route 3: Don't Change the Grid, Change the Distribution — AWQ and SmoothQuant
The third route exploits a beautiful free identity hiding inside matrix multiplication:
Y = X · W = ( X / s ) · ( s · W ) with a per-channel choice of s
Multiply the weights by s, divide the activations by s —
the output is unchanged, but you have reshaped BOTH distributions.
AWQ uses it to protect weights [3]: observe activation magnitudes on a calibration set and identify the weight channels that multiply large activations (~0.1–1%, called salient weights). Note the twist: a weight's importance is judged not by its own magnitude but by its input's magnitude — the same insight as GPTQ's Hessian, in cheaper clothing. Experiments show that keeping just this 1% in FP16 removes most of the quantization damage; but mixed precision is hardware-unfriendly (kernels must juggle two formats), so AWQ instead applies the identity: scale the salient channels up by s > 1 before quantizing. Enlarged, they span more grid points and their relative quantization error shrinks, while the 1/s pushed onto the activations is nearly free. And s isn't hand-picked — it's grid-searched (as activation-magnitude to the power α), selecting the α that minimizes output error.
SmoothQuant applies the same identity in the opposite direction [5]: its target is W8A8 (weights and activations in INT8, unlocking INT8 Tensor Core throughput) — and the obstacle is precisely the activation outliers. Since weights are well-behaved and have "quantization headroom," divide the activations by s to flatten their outliers and migrate that difficulty into the weights (multiplied by s — their distribution gets uglier but survives):
s_j = max|X_j|^α / max|W_j|^(1−α) α ≈ 0.5
α is the migration-strength knob:
α=0 leaves all difficulty in activations; α=1 dumps it all on weights;
0.5 = balanced
AWQ and SmoothQuant make a perfect contrast pair: one identity, applied rightward (amplify salient weights, protect weight-only quantization) and leftward (flatten activation outliers, enable W8A8). They point in opposite directions because they target different deployments — a difference that becomes a hard fork at the kernel level, as we'll see below.
Route 4: Make the Grid Match the Distribution — QLoRA's NF4
The first three routes all stay on a uniform grid. NF4 (NormalFloat-4) attacks the premise itself: who says grid points must be evenly spaced? Weights are approximately normal, so the information-theoretically optimal 4-bit grid should make every grid point equally likely to be used — i.e. take the quantiles of a standard normal distribution as the grid: dense near 0, sparse in the tails [4]. The 16 levels are no longer uniform; quantization becomes a table lookup (snap x/s to the nearest table value), and so does dequantization.
QLoRA stacks two more engineering tricks on top [4]: double quantization — group-wise quantization produces piles of FP32 scales, so quantize the scales themselves to 8-bit, saving ~0.37 bits per parameter on average; and paged optimizers — during fine-tuning, optimizer state pages between GPU and CPU via CUDA unified memory to survive memory spikes. Stacked together: fine-tuning a 65B model on a single 48 GB GPU (base weights frozen in NF4; only the attached LoRA low-rank matrices train; NF4 weights are dequantized to BF16 on the fly for each forward/backward pass). This also pins down NF4's role: it is a storage format designed for frozen, repeatedly-read weights — not a compute format designed for speed.
Deployment Format: GGUF and the k-quants
Theory done — what's inside that .gguf file you actually downloaded? GGUF is the model container of the llama.cpp ecosystem [9][10]: a single file holding weights plus all metadata (tokenizer, hyperparameters, stored as key-value pairs), designed to be mmap-ed — the OS maps the file straight into the address space and loads pages on demand, so an 8 GB model "opens" in seconds and multiple processes can share one read-only copy of the weights.
The Q4_K_M suffix in the filename decodes into exactly the concepts built up above:
Q4_K_M
│ │ └ M = the file's "recipe": different layers get different bit-widths
│ │ (quantization-sensitive layers get more bits)
│ └── K = k-quant: super-block structure
│ 256 weights per super-block, split into 8 sub-blocks of 32
│ each sub-block has its own scale — and the sub-block scales
│ are themselves quantized (same idea as double quantization)
└───── Q4 = 4-bit core, ~4.5 bits/weight effective (scales amortized)
Newer quantizations also use an imatrix (importance matrix): run calibration text, estimate per-weight importance, and weight the quantization error accordingly — recognizably the GPTQ/AWQ "output-error view," simplified and landed inside llama.cpp. So one Q4_K_M file = group-wise quantization + per-layer mixing + re-quantized scales + importance weighting: every brick was laid earlier in this article.
The Moment It Runs: What Actually Happens in the Kernel
How a quantized model gets fast splits into two completely different kernel shapes depending on what was quantized — the single most-confused point in this whole topic:
Weight-only (W4A16 — home turf of GPTQ/AWQ/GGUF): weights sit in memory as 4-bit, but the arithmetic is still FP16. The kernel streams 4-bit weights into registers → dequantizes in place (unpack bits, multiply by s) → multiply-accumulates against FP16 activations. Wait — extra dequantization work makes it faster? Recall section one: decode is memory-bound. The ALUs were idling waiting for data anyway; cutting traffic to 1/4 cuts the waiting to 1/4, and the extra dequant instructions hide inside the formerly idle cycles — nearly free. Same logic on CPU: llama.cpp's block sizes (32/256) are designed around the width of SIMD instructions (AVX2/NEON — single instruction, multiple data), so an integer dot product chews a whole block at once [10].
Weight + activation (W8A8 — home turf of SmoothQuant): both sides in INT8, unlocking the GPU's INT8 Tensor Cores at twice FP16 throughput. This path wins in compute-bound regimes: prefill (ingesting a long prompt at once) and large-batch serving.
scenario bottleneck pick
──────────────────────────────────────────────────────────
local single-user decode memory bandwidth W4A16 (GGUF/GPTQ/AWQ)
long-prompt prefill compute W8A8 / FP8
high-concurrency serving compute+memory W8A8 / FP8 (native HW)
fine-tuning on consumer GPU VRAM capacity NF4 + LoRA (QLoRA)
And to close an earlier loop: besides weights, the KV cache (the attention key/value cache that accumulates token by token during decode) balloons to weight-scale size in long conversations — and it can be quantized to 8-bit or even 4-bit with the exact same scale/group math, just aimed at a different target.
Measuring the Cost, Making the Choice
Quantization is not free. The usual loss metric is the change in perplexity, and the empirical pattern is: 8-bit is essentially lossless; 4-bit with a good method (GPTQ/AWQ/k-quants) loses very little; below 3-bit quality falls off a cliff — the degradation is not linear [2][3]. Larger models also tolerate quantization better: more stable outlier structure, more parameter redundancy. That yields a practical iron rule: at a fixed memory budget, "bigger model + 4-bit" almost always beats "smaller model + FP16." On an 8 GB card, run 7B-Q4, not 3B-FP16.
One more level to keep separate: everything above is PTQ (post-training quantization — compress after training). The other road is QAT (quantization-aware training — simulate quantization during training, using a straight-through estimator to give the non-differentiable round a usable gradient) [8]. QAT is more accurate but requires retraining, which is expensive enough that in practice only model vendors do it (e.g. natively FP8-trained models [6]). Finally, quantization, distillation (training a small model to imitate a big one) and pruning (zeroing out and removing unimportant weights) are three complementary, not competing, axes: distillation cuts parameter count, pruning cuts connections, quantization cuts bits per parameter — and they stack.
The Chain, Tied Together
The whole article in one causal chain: LLMs won't run because weights overflow VRAM and decode is memory-bandwidth-bound → cutting bits per weight attacks both at once → uniform quantization's error bound is s/2, and s is hostage to the max → so group-wise quantization shrinks each s's jurisdiction → but LLM activations grow systematic outliers that blow s up → hence four ways around: isolate (LLM.int8() mixed precision), compensate (GPTQ second-order correction), migrate (AWQ/SmoothQuant scale identity), reshape the grid (NF4 normal quantiles) → GGUF assembles these bricks into a single mmap-able deployment file → at the kernel level the road forks by bottleneck: memory-bound → weight-only, compute-bound → W8A8/FP8 → and the selection rule: at equal memory, big model + 4-bit beats small model + high precision.
Where to Go Next
A few natural next steps — pick one to go deeper:
- Native FP8/FP4 training: when the hardware grid is smart enough, training itself goes low-precision and quantization stops being post-processing [6].
- KV-cache quantization and PagedAttention: in the long-context era the real memory hog is the KV cache, not the weights.
- The ultra-low-bit frontier: 2-bit and even ternary weights (the BitNet line) probing where the cliff really is.
- Speculative decoding: sidestep the memory-bound wall with a small draft model instead of lower precision — orthogonal to quantization, and stackable with it.
References
[1] Dettmers, T. et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. — https://arxiv.org/abs/2208.07339
[2] Frantar, E. et al. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. — https://arxiv.org/abs/2210.17323
[3] Lin, J. et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. — https://arxiv.org/abs/2306.00978
[4] Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. — https://arxiv.org/abs/2305.14314
[5] Xiao, G. et al. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. — https://arxiv.org/abs/2211.10438
[6] Micikevicius, P. et al. (2022). FP8 Formats for Deep Learning. — https://arxiv.org/abs/2209.05433
[7] Nagel, M. et al. (2021). A White Paper on Neural Network Quantization. — https://arxiv.org/abs/2106.08295
[8] Jacob, B. et al. (2017). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. — https://arxiv.org/abs/1712.05877
[9] ggml-org. GGUF File Format Specification. — https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
[10] ggml-org. llama.cpp: LLM inference in C/C++. — https://github.com/ggml-org/llama.cpp