First, unpack “open-source model”: downloadable weights are not the same as open source
Engineers often call every model downloadable from Hugging Face “open source,” but that collapses three different questions: can you obtain the weights, what does the license let you do, and can you reconstruct and modify the training system?
The Open Source Initiative's Open Source AI Definition 1.0 (OSAID) requires the freedom to use, study, modify, and share a system for any purpose. Exercising those freedoms requires more than final parameters: the preferred form for modification includes sufficient data information to build a substantially equivalent system, the complete data-processing and training code, and freely available parameters [1]. This article therefore uses two labels:
- Open source/open science: weights, training and inference code, and data or sufficiently complete data information are available at a level that permits study, modification, and reconstruction. OLMo and Pythia are among the closest examples.
- Open weight: checkpoints are downloadable, usually with inference code and a technical report, but at least one of the training data, complete recipe, or licensing freedoms is missing. Most models popularly called open-source LLMs—including Llama, Gemma, and Kimi—belong here.
This distinction has operational consequences. If a factory embeds a model in an MES incident workflow, weight availability answers “can it run inside the network?” The license answers “may it be used commercially, modified, and redistributed?” Training transparency answers “can its biases and capability origins be audited, and can it be rebuilt?” One checkbox cannot answer all three.
The comparison later in the article therefore separates model license from training transparency. Even Apache-2.0 weights do not make the whole AI system OSAID-compliant when data information and training code are absent; a copyright license on a checkpoint cannot substitute for the missing preferred form of modification.
What they all do: turn “the next token” into a repeated conditional-probability calculation
Llama, Qwen, DeepSeek, Mistral, and Gemma look like different products, but most are variants of the same decoder-only Transformer. “Decoder-only” does not mean a decoder button. It means every position may read only tokens already present to its left and is trained to predict the next token. It drops the original Transformer's separate encoder → cross-attention translation path while retaining causal self-attention derived from scaled dot-product and multi-head attention [2].
One forward path looks like this:
text
│ tokenizer: string → token IDs
▼
[1824, 91, 6207, ...]
│ embedding lookup: use each ID as a row index
▼
X ∈ R^(sequence × hidden)
│
├─ repeat L Transformer blocks ─────────────────────┐
│ │
│ RMSNorm → Q/K/V → RoPE → causal attention ─┐ │
│ ▲ │ │
│ └──────── residual add ◄───────────────┘ │
│ RMSNorm → dense SwiGLU or routed MoE ───────┐ │
│ ▲ │ │
│ └──────── residual add ◄───────────────┘ │
│ │
└───────────────────────────────────────────────────┘
│ final norm + vocabulary projection
▼
logits: one unnormalized score per vocabulary item
│ softmax + decoding policy
▼
next token → append to the input → repeat
The family difference is rarely “does it use a Transformer?” It is which component in this diagram was replaced: whether K/V heads are shared, whether attention is global or sliding-window, how position is encoded, whether the FFN is dense or an MoE, how the router balances experts, how data is filtered, and whether post-training uses a preference loss or reinforcement learning with verifiable rewards. We first need the shared engine at the bottom level before those forks make sense.
Layer one: the tokenizer and embedding define the model's “words”
The network never consumes a Unicode string directly. A tokenizer maps text to integer IDs in a finite vocabulary. Byte Pair Encoding (BPE) starts with small units and repeatedly merges adjacent pairs that co-occur frequently, so common fragments occupy one token while rare strings fall back to several subwords [3]. SentencePiece can learn a BPE or unigram model directly from raw sentences without an English whitespace splitter or a separate Chinese segmenter [4].
This creates three bottom-level trade-offs:
- A larger vocabulary usually shortens sequences but enlarges the embedding and output matrices. With
Vtokens and hidden widthd, the embedding alone has roughlyV × dparameters. An untied output projection adds a similarly sized matrix. - A smaller vocabulary decomposes rare strings more flexibly but lengthens the sequence. Attention prefill cost rises with token count. If the same Chinese SOP takes more tokens than its English version, a nominal 128K context fits different amounts of source text.
- Token boundaries change the learning problem. If
RC0402is consistently represented by a few tokens, learning part-number regularities is easier than when it fragments inconsistently across many positions. This is not one “Chinese ability” knob; it is an artifact of tokenizer, language mixture, and capacity together.
Embedding lookup does not feed the integer 1824 as a continuous number. It treats 1824 as a row index into a trainable matrix E ∈ R^(V×d). Initial rows contain no dictionary definitions. Backpropagation under next-token loss gradually moves features that reduce error in related contexts into useful geometry. Lookup is the mechanism, cross-entropy is the training pressure, and the semantic space is the artifact. Conflating them obscures where capability comes from.
Layer two: the residual stream is the information trunk
Each token's hidden vector travels through all blocks along a residual stream. A common pre-norm block is:
h' = h + Attention(RMSNorm(h))
h'' = h' + FFN_or_MoE(RMSNorm(h'))
A residual addition means a sublayer writes an update instead of replacing all prior state. Gradients also retain a near-identity path toward earlier layers, so information does not have to survive a long series of nonlinear bottlenecks.
RMSNorm—Root Mean Square Layer Normalization—computes:
rms(x) = sqrt((1/d) · Σ_i x_i² + ε)
RMSNorm(x)_i = g_i · x_i / rms(x)
d is hidden width, ε prevents a near-zero denominator, and g_i is a learned scale. Unlike LayerNorm, RMSNorm does not subtract the mean. The paper hypothesizes that re-centering invariance is dispensable while retaining re-scaling invariance [5]. Its efficiency follows from computing fewer statistics and operations, not from the label “RMS”; the corresponding trade-off is that it does not provide LayerNorm's invariance to shifts in the mean.
Layer three: how self-attention actually moves information between tokens
The model forms three linear projections from the current hidden states X:
Q = XW_Q Query: what this position is looking for
K = XW_K Key: under what condition this position can be found
V = XW_V Value: what content is moved when it is selected
scores = QKᵀ / sqrt(d_head)
weights = softmax(scores + causal_mask)
context = weights · V
QKᵀ is the dot product between every query and key. As head width d_head grows, the variance of random dot products grows. Dividing by sqrt(d_head) brings them back to a stable scale and keeps softmax away from a prematurely saturated, low-gradient region [2]. The causal mask assigns negative infinity to future positions, making their post-softmax weight effectively zero.
Softmax is more than “normalization”:
softmax(z_i) = exp(z_i - max(z)) / Σ_j exp(z_j - max(z))
Subtracting max(z) is numerically stable and leaves probability ratios unchanged. Exponentiation preserves ordering while amplifying differences; every result is positive and sums to one. The context is therefore a data-dependent weighted sum of values, not a hard database lookup. Multiple positions usually retain nonzero weight, and a high attention weight is not automatically a faithful causal explanation.
Multi-head attention splits hidden space into parallel Q/K/V subspaces. Training may make different heads useful for different relationships, after which their outputs are concatenated and projected. No head is manually guaranteed to be a “syntax head” or “part-number head.” Specialization is a learned artifact, not an architectural promise.
RoPE: rotate Q and K instead of adding a position vector to the token
Without position information, content dot products alone cannot adequately distinguish reordered copies of the same tokens. Rotary Position Embedding (RoPE) treats neighboring Q/K coordinates as a 2-D plane and rotates them by position-dependent frequencies [6]:
R(mθ) = [ cos(mθ) -sin(mθ) ]
[ sin(mθ) cos(mθ) ]
q_m' = R(mθ) q_m
k_n' = R(nθ) k_n
(q_m')ᵀ k_n' = q_mᵀ R((n-m)θ) k_n
m and n are positions. The final dot product depends on relative displacement n-m, even though each vector was rotated using its absolute position. That is the bottom-level meaning of “relative position from absolute rotations.”
Changing the RoPE base, applying position interpolation, or configuring YaRN does not magically teach reliable long-document reasoning. YaRN combines frequency-aware interpolation/scaling, an attention-magnitude adjustment, and continued training to extend context [11]. It addresses positional distribution outside the original training window. Whether a model can join evidence at token 7K and token 91K still depends on long-sequence data, tasks, and learned attention behavior. Accepted input length is an interface limit; distant-evidence utilization is a trained artifact.
MHA, MQA, GQA, and the KV cache: the practical difference appears during decode
When generating token t+1, the K and V tensors for the preceding t tokens do not change. An inference engine stores them in a KV cache rather than recomputing the prompt every step. Approximate cache capacity per batch item is:
KV bytes ≈ 2 × layers × sequence × KV_heads × head_dim × bytes_per_element
↑
K and V
Multi-Head Attention (MHA) assigns each query head its own K/V head. Multi-Query Attention (MQA) shares one K/V set across all query heads. Grouped-Query Attention (GQA) sits in between: a group of query heads shares one K/V set [7].
MHA: Q0→K0,V0 Q1→K1,V1 Q2→K2,V2 Q3→K3,V3
GQA: Q0,Q1→K0,V0 Q2,Q3→K1,V1
MQA: Q0,Q1,Q2,Q3→K0,V0
Holding query heads constant, reducing 64 KV heads to 8 makes the head dimension of the cache roughly one eighth as large. That directly lowers long-conversation decode memory and bandwidth. The cost is that more queries share fewer key/value representations, potentially reducing capacity. GQA was designed to approach MHA quality with MQA-like efficiency [7].
Two mistakes are common. First, GQA does not shrink all model weights by eight; it changes the K/V projections and cache. Second, FlashAttention is not a different learned attention mechanism. It tiles the same exact calculation in SRAM, reducing HBM traffic and avoiding materialization of the full S×S matrix [10]. That is an execution-kernel change, unlike GQA or sliding-window attention.
FFN, SwiGLU, and MoE: attention moves information; the FFN rewrites features
Attention lets positions read one another. The Feed-Forward Network (FFN) applies a nonlinear transformation independently at each position. Many modern LLMs use SwiGLU:
gate = SiLU(XW_gate)
up = XW_up
FFN(X) = (gate ⊙ up) W_down
SiLU(x) = x · sigmoid(x)
⊙ denotes elementwise multiplication. One branch supplies content and the other a gate, allowing input-dependent feature passage instead of a single projection → activation → projection. Experiments on GLU variants found that such gated FFNs can improve Transformer quality [8].
A dense model sends every token through the same FFN parameters. A Mixture of Experts (MoE) replaces that FFN with many experts and a top-k router:
router scores r = softmax(W_r x)
selected = top_k(r)
y = Σ_(e in selected) normalized_r_e · Expert_e(x)
Each expert is usually its own FFN. With 256 experts and 8 selected per token, total capacity can be enormous while per-token arithmetic activates only a small fraction. That is how total parameters and active parameters become decoupled [9].
MoE is not free scale:
- Every expert's weights must live somewhere in cluster memory. Active parameters are not checkpoint size.
- Router decisions require all-to-all token dispatch across devices. With small batches, experts remain under-filled and communication/kernel overhead can erase FLOP savings.
- A router may overload a few experts while starving the rest. Traditional load-balancing auxiliary losses fight this, but a strong auxiliary objective can force semantically unnatural routing and conflict with language loss.
- Top-k routing is chosen per token and layer. A sentence is not assigned once to a fixed “math expert.” Expert specialization is a trained artifact.
These are the problems that separate Mixtral's top-2 sparse MoE, DeepSeekMoE's fine-grained and shared experts, DeepSeek-V3's auxiliary-loss-free balancing, and Qwen's ultra-sparse MoE.
From logits to text: the model predicts a distribution; decoding picks one path
The final hidden state is projected to V logits. A logit is an unnormalized real score, not a confidence percentage. Softmax gives:
p(token_i | prefix) = exp(logit_i / T) / Σ_j exp(logit_j / T)
T is temperature. Values below one amplify logit differences; values above one flatten them. Greedy decoding chooses the maximum, top-k samples among the highest k, and top-p samples from the smallest set whose cumulative probability reaches a threshold. These are inference policies. Changing temperature does not modify weights or teach the model another fact.
Generation is sequential: select one token, append it to the prefix, and run another step. During training, teacher forcing lets all positions' next-token losses be computed in parallel because the true prefix is known. During inference, output token 101 depends on the 100 choices already made, so unknown future tokens cannot all be generated in parallel. This is why training and prompt prefill can saturate a GPU while decode is often constrained by rereading weights and cache each step.
The three layers most often confused: architecture, objective, and artifact
┌──────────────────────────────────────────────────────────────┐
│ Mechanism / architecture │
│ RoPE, GQA, MLA, SWA, SwiGLU, MoE, residual, KV cache │
│ Question: how does information flow, compute, and memory grow?│
├──────────────────────────────────────────────────────────────┤
│ Training objective │
│ next-token CE, SFT CE, DPO, reward model + PPO, GRPO │
│ Question: which output lowers scalar loss or raises reward? │
├──────────────────────────────────────────────────────────────┤
│ Artifact / behavior │
│ weights, representation, instruction following, tool use, │
│ and long-reasoning habits │
│ Question: what did data, objectives, and optimization produce?│
└──────────────────────────────────────────────────────────────┘
“DeepSeek-R1 reasons, therefore it must contain reasoning attention” is a layer error. Its backbone inherits DeepSeek-V3; long reasoning behavior primarily comes from post-training data, verifiable rewards, and RL pressure. Conversely, MoE does not inherently produce tool calling. Tool schemas, decisions to call, and result handling are normally shaped by SFT/RL data and the chat template.
The complete workflow for training a modern LLM
1. Acquisition, cleaning, and data mixture
Sources may include web pages, books, code, papers, mathematics, dialogue, and synthetic material. “15T tokens” alone says little about quality. Source ratios, language mix, deduplication, quality classifiers, PII/safety filters, contamination controls, and curriculum determine the distribution repeatedly applied to the weights.
Scaling-law work found power-law relationships between cross-entropy loss, model size, data, and compute in the studied ranges [12]. Chinchilla then showed that under its fixed-compute setting, parameters and training tokens should scale roughly together; a smaller, sufficiently trained model can beat a larger undertrained one [13]. Many modern open models train relatively small networks for several to tens of trillions of tokens because the model will be served repeatedly: more one-time pretraining can buy cheaper lifetime inference. That is deployment economics, not a law that more parameters are always smarter.
2. Pretraining: next-token cross-entropy
For a token sequence x_1...x_T, the standard base-model loss is:
L_NTP = - Σ_t log p_θ(x_t | x_<t)
The lower the probability assigned to the actual next token, the larger the negative-log penalty. Backpropagation computes each parameter's gradient; the optimizer applies learning rates, momentum, and numerical-stability machinery to update weights. The result is a base continuation model, not automatically a reliable assistant. Web data includes answers, headings, ads, arguments, errors, and inconsistent roles.
Some families add Multi-Token Prediction (MTP): shared representations also predict several later tokens. This supplies denser future supervision, but normal inference is still autoregressive. MTP is a training objective and potentially a speculative-decoding aid, not fully parallel ordinary generation.
3. Continued pretraining: change domain or context distribution
Code, math, manufacturing documents, or long-context capability can be strengthened by continuing next-token training from a base checkpoint. This remains a pretraining-style objective with a changed mixture or length distribution. It differs from SFT not in whether gradients exist, but in whether data consists of prompt→desired-response demonstrations and whether loss is concentrated on response tokens.
4. SFT: demonstrate how the model should respond
Supervised Fine-Tuning (SFT) places system/user/assistant conversations into a chat template and applies cross-entropy to ideal assistant responses. It teaches format, tone, refusals, tool schemas, and reasoning traces. The canonical InstructGPT pipeline used human demonstrations for SFT, pairwise rankings to train a reward model, and then RLHF [14].
SFT is imitation. Strategies not represented in demonstrations are difficult to discover through imitation alone, and low-quality long traces teach verbosity and errors. Preference optimization and reinforcement learning address that limitation.
5. Preference optimization: learn that A is better than B
Classic RLHF samples several answers to one prompt, collects human or AI rankings, fits a reward model, then optimizes the policy for reward while imposing a KL penalty against a reference model [14]. KL divergence measures distributional drift. Without a constraint, the policy may exploit reward-model blind spots instead of becoming genuinely better.
Direct Preference Optimization (DPO) derives a pairwise classification objective from the optimum of KL-constrained RLHF. It directly increases the chosen response's log-probability margin over the rejected response, without a separate reward model and online PPO loop [15]. It is simpler and stable, but remains limited by preference-pair quality and does not explore solutions absent from the data.
6. Reasoning RL: use verifiable outcomes to push search behavior into the policy
Math answers, unit tests, format constraints, and tool environments can provide relatively objective rewards. Group Relative Policy Optimization (GRPO) samples a group of completions for one prompt and estimates advantage relative to the group's rewards, eliminating PPO's separately learned critic/value model [16]. DeepSeek-R1-Zero showed that large-scale RL without preliminary SFT can produce long reasoning behavior, but with readability and language-mixing problems. R1 therefore adds cold-start data, multiple RL/SFT stages, and distillation.
RL does not bolt a symbolic theorem prover onto the Transformer. It repeats sampling → scoring → increasing the probability of high-reward token trajectories. Reward hacking remains possible, and open-ended claims without a verifier can still be confidently wrong. A long chain of thought is not proof that every sentence faithfully exposes the internal causal process.
7. Distillation and synthetic data: move a larger model's distribution into a smaller one
A student can learn a teacher's logits, final answers, or reasoning traces. Phi and Gemma particularly emphasize high-quality synthetic data or teacher distillation. This gives a compact model denser, cleaner supervision than raw web text on a targeted distribution. It can also reproduce the teacher's errors, style, and blind spots. Merely disclosing “synthetic data” is not enough to rebuild the generation prompts, filters, and mixture.
The causal chain so far is:
architecture defines the representable, computable, deployable space
↓
data defines the distribution repeatedly shown to the network
↓
loss / reward defines which errors receive pressure
↓
optimization accumulates that pressure into weights
↓
decoding turns a conditional distribution into one realized output
That is why models cannot be compared by parameter count or benchmark score alone. We can now inspect each family and identify the layer in which it actually made a different choice.
Llama: the dense compatibility baseline, not a synonym for open source
Llama 3 is the cleanest baseline for modern dense decoders. Its text models use a 128,256-entry tokenizer, RMSNorm, RoPE, SwiGLU, and Grouped-Query Attention. The 8B, 70B, and 405B variants keep the same broad computation graph while scaling depth and width; every token activates every layer's attention and FFN parameters [17]. That makes total parameters a reasonable approximation of active parameters, unlike sparse MoE models.
Meta reports more than 15T pretraining tokens for Llama 3 and a 128K-context extension for Llama 3.1. Post-training combines instruction data, reward modeling, rejection sampling, SFT, and preference/RL methods [17]. None of those add a new attention edge: they change which token trajectories receive gradient or reward pressure. The same dense backbone can therefore exist as Base, Instruct, or chat-safety variants with very different behavior.
The engineering advantage follows from regularity. Dense routing is predictable, and the Llama layout has broad support in vLLM, llama.cpp, GGUF, quantizers, and fine-tuning tools. The disadvantage also follows from regularity: a 405B token activates the entire model, while an MoE can expose larger conditional capacity at lower per-token arithmetic. Llama uses a custom Community License with acceptable-use terms and a large-service clause; downloadable weights are therefore open weight under custom terms, not OSAID Open Source AI [18].
Mistral and Mixtral: sliding windows and sparse experts attack different bottlenecks
Mistral 7B combines GQA with Sliding-Window Attention (SWA) [19]. With window W, a token at position t attends directly only to roughly [t-W+1, t]. Cache can be implemented as a rolling buffer, so retained K/V per layer is bounded by the window rather than the full sequence. Stacking layers still enlarges the receptive field because a current token can read a nearby hidden state that already incorporated earlier neighbors.
layer 1: direct attention covers one recent window
layer 2: neighbors already summarize their previous windows
layer N: information can propagate farther across layers
benefit: bounded per-layer attention/cache
cost: no direct arbitrary-position edge to every old token in every layer
Mixtral 8x7B keeps the attention backbone but replaces each dense FFN with eight experts; the router selects two per token. It has about 47B total parameters but roughly 13B active per token, with 32K context and Apache-2.0 weights [20]. “8x7B” does not mean eight independent 7B models because embeddings and attention are shared. Sparse arithmetic does not remove the need to store all experts, and distributed inference adds token dispatch and all-to-all communication.
The 2026 Mistral Small 4 shows how far that line progressed: 119B total, about 6.5B active, 128 experts with top-4 routing, up to 256K context, and Apache-2.0 weights [21]. Its extremely low active fraction is attractive only when the runtime can keep expert routing, batching, and communication efficient. Mistral has not disclosed complete pretraining token totals and mixture recipes for these representatives, so benchmark strength cannot be reverse-engineered into an invented data number.
Qwen: dense-to-MoE coverage plus a post-trained thinking policy
Qwen3 spans small dense checkpoints through 30B-A3B and 235B-A22B MoE models. Its report describes roughly 36T pretraining tokens across 119 languages, code, mathematics, and long-context stages [22]. In the MoE variants, shared experts supply general capacity while routed experts specialize; a token sees the shared path plus only selected routed experts. The A3B or A22B suffix is active parameters, not download size.
Qwen3's thinking/non-thinking behavior belongs mainly to post-training. /think and /no_think, together with the chat template, steer one weight set toward long reasoning trajectories or direct responses. The reported pipeline uses long-chain-of-thought cold start, reasoning RL, thinking-mode fusion, and general RL [22]. The mechanism is not a hidden symbolic engine: control tokens change the conditioning context, and training has made different trajectory distributions likely under those conditions.
same backbone weights
│
├─ /think → policy favors longer reasoning trajectories
└─ /no_think → policy favors direct, lower-latency answers
the branch was carved by data/reward; it is not a second attention network
Qwen3-Next 80B-A3B pushes efficiency further with many experts and a hybrid pattern of gated delta-style linear attention and periodic full attention; its configuration supports about 262K positions [24]. Recurrent state compresses history instead of retaining a fully addressable KV record at every layer. Periodic full attention repairs exact token-to-token access. Qwen3.5/3.6 move to native early-fusion multimodality, so this text-only deep dive uses Qwen3 and Qwen3-Next as the comparable text computation graphs. Qwen3 checkpoints are released under Apache-2.0 [23], but the unavailable corpus and complete training pipeline still prevent strict whole-system reproducibility.
DeepSeek: MLA, fine-grained MoE, MTP, reasoning RL, then content-sparse attention
DeepSeek is useful because it changed several different layers of the stack. MLA and DeepSeekMoE are architecture choices; auxiliary-loss-free balancing and Multi-Token Prediction affect training; GRPO and verifiable rewards affect post-training; the V3, R1, and V3.2 artifacts therefore should not be collapsed into one label.
MLA compresses K/V into a learned latent cache
Multi-head Latent Attention first down-projects each token's hidden state into a compact latent c_t^KV. In inference, that latent plus a decoupled positional component is cached; learned up-projections reconstruct content keys and values for the heads [25]. RoPE is separated because directly rotating a compressed shared latent would obstruct the algebraic weight absorption used by efficient implementations.
h_t ── W_DKV ──► c_t^KV (compact cached latent)
├── W_UK ─► content keys
└── W_UV ─► values
h_t ── separate RoPE projection ─► positional key component
GQA reduces cache by sharing discrete KV heads. MLA instead learns a low-rank latent from which head-specific content is reconstructed. Both target decode-time cache, but with different kernels and approximation constraints.
DeepSeekMoE, balancing bias, and MTP
DeepSeek-V3 has about 671B total and roughly 37B active parameters, with 256 routed experts, eight selected per token, plus a shared expert. It was pretrained on 14.8T tokens [26]. Fine-grained experts increase combinatorial specialization but require expert parallelism and balanced routing. V3's auxiliary-loss-free method updates a per-expert bias used in top-k selection when experts are over- or under-utilized, while leaving the semantic affinity used to weight outputs separate. This reduces the need for a large balancing loss that could fight language modeling, although a small sequence-level auxiliary loss remains [26].
Multi-Token Prediction (MTP) adds training modules that predict more than the immediate next token from shared representations. The deployed main model remains autoregressive, but training provides denser future-token supervision; the auxiliary module can also support speculative decoding [26]. MTP does not mean the production model independently emits several guaranteed-correct tokens at once.
R1 changes the policy; V3.2 changes long-context attention
R1-Zero starts from V3-Base and applies GRPO with verifiable rewards without conventional preliminary SFT. Long reasoning emerges, but readability and language mixing suffer. R1 adds cold-start reasoning examples, reasoning RL, rejection-sampled SFT, and a later alignment RL stage; it also distills generated reasoning data into dense Qwen/Llama students [27]. A 32B distilled checkpoint is therefore a Qwen-architecture student trained on R1 traces, not a compressed 671B MoE with MLA.
DeepSeek-V3.2 adds DeepSeek Sparse Attention (DSA). A lightweight “lightning indexer” scores historical latent KVs, selects top-k positions, and lets expensive MLA attend only to those positions [28]. The main attention path falls from quadratic pair scoring toward O(Lk), although the lightweight indexer still examines sequence-wide candidates. This is content-based sparsity, unlike Mistral's distance-based fixed window.
all historical latent KVs
│ lightweight indexer produces coarse relevance
▼
top-k positions
│ MLA computes expensive attention only there
▼
context update
DSA can jump to a distant part-number definition, but a missed candidate receives no expensive second look. V3.2 adds specialist distillation, mixed GRPO, and agent environments [28]. Older V3 weights used the DeepSeek Model License, while R1 and V3.2 publish MIT checkpoints; distilled models retain their base-family obligations. Licensing must therefore be checked per checkpoint, and the missing training corpus/full code still places the system in the open-weight rather than strict OSAID category.
Gemma: local/global attention and distillation as small-model engineering
Gemma is Google's open-weight decoder family, not simply a “smaller Gemini.” Gemma 2 uses RMSNorm, RoPE, GQA, and GeGLU, alternating local sliding-window and global attention. Its 9B and 27B models also learn from larger-teacher logits through knowledge distillation [29]. GeGLU and SwiGLU are both gated FFNs; GeGLU uses GELU for the gate while SwiGLU uses SiLU.
Gemma 3 changes the text pattern to five local layers per global layer, with a 1,024-token local window, global context up to 128K, and RMS normalization on queries and keys [30]. A local layer cannot directly retrieve a definition 40K tokens earlier, but a preceding global layer can mix that definition into a current hidden state for subsequent local processing. The savings and limitation follow from the same mechanism: most layers are cheap, while exact long-range interaction does not occur in every layer.
local → local → local → local → local → GLOBAL
│ mix within a 1,024-token neighborhood │
└─────────────────────────────────────────────────────┘
the global layer refreshes long-range alignment
Gemma 2 reports about 2T/8T/13T tokens for 2B/9B/27B and mixes distillation with next-token training; Gemma 3 adds long-context, multilingual, and synthetic-data stages [29][30]. Small-model strength therefore comes from architecture, data selection, teacher distribution, and post-training together. Gemma 1–3 use custom Gemma Terms. The 2026 Gemma 4 release moves new checkpoints to Apache-2.0 and includes early-fusion multimodal and MoE variants; this text-only article notes that license/family transition without treating its vision path as a text architecture [31].
Phi: a 14B dense graph whose main bet is curriculum and synthetic data
Phi-4 is a 14B dense decoder with 40 layers, hidden size 5,120, 40 query heads and 10 KV heads, FFN size 17,920, 16K context, and a 100,352-token vocabulary [32]. Four query heads share each KV group. There is no router or latent KV reconstruction, so every token follows the same weights and latency is regular.
The distinctive choice is mostly at the training layer. The technical report describes about 9.8T tokens combining public web, code, academic material, and substantial synthetic data, scheduled through a data curriculum, followed by SFT and DPO [32]. A teacher generates exercises, solutions, or rewrites; filters reject errors and duplicates; cross-entropy then trains the student. This creates dense supervision for rare reasoning patterns, but also transfers teacher errors, generation-template bias, and filter blind spots. Phi-4 is MIT licensed, while the complete prompts, mixture, and cleaners remain unavailable: permissive open weights are not a reproducible training system.
OLMo: making model formation auditable is itself the contribution
Ai2's OLMo line releases data, training code, intermediate checkpoints, evaluations, and records, letting researchers investigate which stage produced a behavior [33]. In the three-layer framework, that matters because weights are only the artifact; without mixture, code, state, and checkpoints, one cannot replay how objectives carved the artifact.
OLMo 2 remains a dense decoder but changes normalization placement on attention/MLP output branches, adds QK-norm, and uses z-loss [33]. z-loss penalizes the square of logsumexp(logits), discouraging all unnormalized scores from growing together and improving numerical stability. It is not an instruction-following objective. OLMo 3 extends lifecycle releases across base, SFT, DPO, and RL stages so behavioral changes can be attributed more carefully [34].
Olmo Hybrid 7B (2026) makes the architecture-level change: 32 layers alternate three Gated DeltaNet layers with one full multi-head-attention layer, supporting 65,536 context [35]. Delta-like layers keep a recurrent fast-weight state rather than an individually addressable K/V record for every token:
S_t = decay_t ⊙ S_(t-1) + write_t
o_t = read(q_t, S_t)
3 DeltaNet layers: compress history into fixed-size recurrent state
1 full-attention layer: restore exact token-to-token addressing
The state scales gently with length, but collisions and compression erase exact historical addresses. Periodic full attention is therefore a complement. The report lists 5.5T main-stage tokens followed by 100B and 50B stages. Weights, code, and data chain are broadly open under Apache/open data terms, although some full logs were still marked forthcoming at release; openness must be judged by available artifacts, not branding [35].
GLM: from conventional dense GQA to MLA, DSA, and giant agentic MoE
GLM names several different graphs. GLM-4-9B is a 40-layer dense decoder with hidden size 4,096, 32 query heads/two KV heads, RMSNorm, SwiGLU, and rotary positions. GLM-4-32B-0414 expands the dense line [36]. GLM-4.5 moves to MoE: roughly 355B total/32B active for the flagship and 106B/12B for Air, with MTP, around 23T pretraining tokens, and post-training for reasoning, coding, agents, and thinking/non-thinking behavior [36]. Tool ability comes from trajectories and environment rewards shaping the policy, not a special tool circuit added to inference.
GLM-5/5.2 scales to roughly 744B total/40B active and combines MLA, MoE, parameter-shared MTP, and DeepSeek-style sparse attention. The 5.2 card lists up to 1M context [37][38]. Parameter-shared MTP reuses parameters across future-prediction depths, providing denser future supervision without a separate large head for each depth.
GLM-5.2 also introduces IndexShare: one sparse-attention indexer is shared across roughly four layers [38]. This saves indexer parameters and work and gives neighboring layers consistent candidates; if the indexer misses a crucial token, several layers may share the miss. Muon-like optimization for matrices and AdamW for embedding/norm parameters changes how weights are learned, not the inference attention graph.
query ─► shared indexer ─► top-k token positions ─► MLA attention
▲ │
└──────┴── candidate index reused across ~4 sparse layers
Licensing changes by version: early glm-4-9b-chat used a custom revocable model license, the 0414 32B release used MIT, and GLM-4.5/5 repositories publish Apache or checkpoint-specific files [36][38]. “GLM is open” is not a safe family-wide conclusion.
Kimi K2: MuonClip stabilizes trillion-parameter MoE training
Kimi K2 is a 1.04T-total, roughly 32.6B-active text MoE with 61 layers, 384 routed experts, eight selected per token plus a shared expert, MLA, and 128K context [39]. It performs about active-model arithmetic per token but still stores and distributes a trillion parameters. Router dispatch also creates all-to-all traffic.
Its central pretraining contribution is MuonClip. Muon approximately orthogonalizes matrix-gradient updates, distributing update strength more evenly across singular directions. At extreme scale, Q/K weights can still amplify attention logits. MuonClip monitors their spectral norm—the largest singular value and thus maximum input amplification—and rescales when a threshold is exceeded [39]. This is an optimizer/stability mechanism, not an inference layer.
gradient ─► Muon update ─► Q/K weights
│ spectral norm too large?
├─ yes: rescale
└─ no: keep
↓
bound attention-logit growth
K2 Base used about 15.5T tokens. Agentic SFT and joint RL sample code, math, and tool trajectories and score their outcomes. K2 Thinking integrates long reasoning and tool use; its card supports 256K and native INT4 artifacts [39][40]. INT4 describes a deployed quantized artifact, not four-bit pretraining. The Modified MIT license adds a Kimi attribution requirement above stated monthly-active-user or revenue thresholds, so it is not standard MIT. K2.5 is multimodal and is excluded from the pure-text graph comparison.
Yi: a conservative dense control showing the power of data and adaptation
Yi 1/1.5 uses a Llama-like dense decoder with RMSNorm, RoPE, GQA, and SwiGLU in 6B, 9B, and 34B sizes. Yi-1.5 follows roughly 3.1T original tokens with about 500B continued-pretraining tokens and roughly 3M instruction samples [41]. With no MoE, MLA, or recurrent attention, its value here is as a control: the same mature parts can produce a different capability profile through bilingual data, filtering, long-context continuation, and alignment.
Yi-200K extends RoPE through long-sequence continued training, but its own report gives conflicting 5B and 10B token counts for that stage. Without an erratum, the conflict should remain visible rather than silently choosing one [41]. Yi weights are Apache-2.0 [42], while complete instruction data and cleaning remain unavailable, making it principally open weight.
The open-science lineage: BLOOM, GPT-NeoX, Pythia, Falcon, MPT, and DBRX
BLOOM is a 176B dense multilingual decoder built through BigScience, with ROOTS corpus work, training logs, and Megatron-DeepSpeed engineering exposed. It demonstrated multinational governance at scale, but BLOOM RAIL restricts some uses and therefore is not unrestricted OSAID open source [43]. GPT-NeoX-20B made a scalable stack and Pile-based training accessible; Pythia fixed data order and released 70M–12B suites with checkpoints every 1,000 steps, enabling controlled studies of scaling and memorization [44].
Falcon linked RefinedWeb deduplication/quality filtering with multi-query attention, illustrating that data engineering and KV efficiency are independent levers [45]. MPT used ALiBi, FlashAttention, and Apache-licensed base weights as an engineering bridge for long context [46]. DBRX's 132B-total/36B-active, 16-expert top-4 design illustrated fine-grained MoE, but its Databricks Open Model License and acceptable-use policy are custom terms; the word “Open” in a license title does not make it OSI-approved [47].
Pythia and OLMo can be more scientifically open than stronger open-weight checkpoints because they publish formation history, not only the final artifact. A capability leaderboard asks which output is better today; open science asks whether the behavior can be traced, tested, and rebuilt. Those are separate axes.
Architecture comparison: compare computation graphs before benchmarks
Each row is a representative checkpoint, not an immutable family property. A means approximate parameters activated per token. Context is a supported maximum, not proof of uniformly reliable evidence retrieval.
Model / checkpoint Dense or MoE Attention / memory Context Fundamental tradeoff
────────────────────────────────────────────────────────────────────────────────────────────────────────────
Llama 3.1 405B Dense GQA + RoPE 128K broad support; full model active
Mistral 7B Dense GQA + sliding window 8K* rolling KV; distance-limited edges
Mixtral 8x7B 47B / A~13B GQA/SWA + top-2 experts 32K low active FLOPs; store all experts
Mistral Small 4 119B / A6.5B granular MoE, top-4 256K ultra-sparse; expert runtime required
Qwen3 235B-A22B 235B / A22B GQA + shared/routed experts 128K dual-mode policy; large total weights
Qwen3-Next 80B-A3B 80B / A3B delta-style + full attention 262K compact state; lossy exact recall
DeepSeek V3/R1 671B / A~37B MLA + DeepSeekMoE + MTP 128K compact latent KV; complex kernels
DeepSeek V3.2 671B / A~37B MLA + DSA top-k token attention 128K content sparse; indexer-recall risk
Gemma 3 27B Dense 5 local : 1 global + QK-norm 128K cheap majority; nonuniform long edges
Phi-4 14B Dense GQA + RoPE 16K regular path; data does heavy lifting
Olmo Hybrid 7B Dense hybrid 3 DeltaNet : 1 full attention 65K fixed state plus periodic exact access
GLM-5.2 744B / A40B MLA + DSA + IndexShare 1M long reach; shared misses propagate
Kimi K2 1.04T / A32.6B MLA + 384 experts, top-8 128K huge capacity; storage/all-to-all cost
Yi-1.5 34B Dense GQA + RoPE 32K/200K simple mature graph; no cache redesign
* Mistral 7B materials distinguish sequence/context settings from a 4K attention window;
deployment must follow the exact checkpoint configuration.
The mechanisms are orthogonal. GQA/MLA mainly attack decode-time KV cache. MoE decouples total capacity from per-token arithmetic. SWA/DSA/DeltaNet alter which historical information reaches the current token. FlashAttention only changes GPU tiling and data movement, not the mathematical attention graph. A model can use MLA, MoE, DSA, and FlashAttention simultaneously.
Training and openness comparison: a weight license is not reproducible training
Family Main disclosed training route Weight/model terms* Data+code+process transparency
────────────────────────────────────────────────────────────────────────────────────────────────────────────
Llama 3 >15T NTP → SFT/RM/RS/DPO/RLHF Llama Community low
Mistral/Mixtral NTP; partial instruct recipe mostly Apache-2.0 reps low; totals/mix undisclosed
Qwen3 36T NTP → cold start → reasoning/general RL Apache-2.0 low-medium
DeepSeek V3/R1 14.8T NTP+MTP → SFT/GRPO/RL/distillation versioned; V3.2/R1 MIT medium
Gemma 2/3 NTP + teacher distillation → SFT/RL Gemma Terms low-medium
Phi-4 9.8T curated/synthetic → SFT+DPO MIT low
OLMo 2/3/Hybrid NTP → SFT/DPO/RL, with staged checkpoints Apache-2.0 high
GLM-4.5/5 23T+ NTP/MTP → reasoning/coding/agent RL Apache/MIT by checkpoint medium
Kimi K2 15.5T NTP+MuonClip → agentic SFT+joint RL Modified MIT medium
Yi-1.5 3.1T + 0.5T continuation → ~3M SFT samples Apache-2.0 low
Pythia fixed-order NTP, multiple scales/steps Apache-2.0 high
BLOOM ROOTS NTP plus public logs BLOOM RAIL high artifacts; use limits
* Read the LICENSE beside the exact downloaded checkpoint. Family names do not freeze legal terms.
Use three separate tests. Can weights be used, modified, and redistributed? That is the license axis. Are data information and training/processing code sufficient to rebuild a substantially equivalent system? That is OSAID's system-openness axis [1]. Are intermediate checkpoints, logs, and data order available for causal replay? That is the open-science axis. Apache weights can score high only on the first. BLOOM can expose unusually rich artifacts while its RAIL restrictions prevent unrestricted open-source classification.
Choosing for RAG, MES, and private deployment
For Oracle/MES queries, part-number explanations, and RAG, first ask whether the runtime correctly supports the attention pattern, quantization, and tokenizer. A single server or small GPU pool usually starts with dense Llama, Qwen, Mistral, Phi, Gemma, Yi, or OLMo sizes. Fixed token paths make memory and latency predictable, while GGUF, vLLM, and quantization support are mature. For data that cannot leave the factory, maintainability can matter more than a few benchmark points.
With a high-throughput multi-node service, DeepSeek, Qwen MoE, GLM, Kimi, and Mixtral can turn conditional capacity into value—but only if batches fill experts and expert-parallel all-to-all does not dominate. Active parameters approximate matrix arithmetic, not download size, GPU memory, or cold-start time.
Long context does not replace retrieval. Stuffing 100K tokens of process documentation into every prompt keeps prefill expensive, and “accepted by the API” does not mean “reliably recovered.” RAG should first narrow evidence through metadata, vector, and lexical retrieval, then let the model reason over a controlled context. GQA/MLA reduce cache and SWA/DSA reduce attention work; they do not fix document permissions, freshness, or citation correctness.
Base, instruction, and reasoning checkpoints are also not interchangeable. A Base model learns continuation and may not obey a chat template. SFT/preference training teaches tools, refusals, and response form. Verifiable-reward RL raises the probability of long successful trajectories but can increase token cost or reward hacking. Validate against your MES schema, error codes, permission boundaries, and abstention rules rather than treating a general benchmark as acceptance testing.
Closing chain: the reusable mental model
text → tokenizer → token IDs → embeddings
→ [norm → attention/memory mixing → residual
→ norm → dense FFN or routed experts → residual] × L
→ logits → decoding → next token
next-token loss carves language and knowledge statistics
SFT carves demonstrated response formats
preference/RL raises the probability of higher-reward trajectories
distillation transfers a teacher distribution or traces to a student
GQA/MLA address KV cache; MoE addresses capacity vs active compute;
SWA/DSA/recurrent hybrids address long-sequence cost;
data, objectives, licenses, and reproducibility are never guaranteed by an architecture name.
A valid comparison therefore asks three layers in order. At the architecture layer: which parameters does each token traverse, what history is addressable, and how does cache grow? At the objective layer: which data and rewards shape the graph? At the artifact layer: did you obtain a Base, Instruct, reasoning, quantized, or distilled checkpoint, and what does its exact license allow? Only then do Llama's ecosystem, Mistral's windows, Qwen's dual policy, DeepSeek's MLA/DSA, OLMo's transparency, and Kimi/GLM's giant MoE become engineering choices rather than brand impressions.
References
[1] Open Source Initiative (2024). The Open Source AI Definition 1.0. — https://opensource.org/ai/open-source-ai-definition [2] Vaswani et al. (2017). Attention Is All You Need. — https://arxiv.org/abs/1706.03762 [3] Sennrich, Haddow & Birch (2015). Neural Machine Translation of Rare Words with Subword Units. — https://arxiv.org/abs/1508.07909 [4] Kudo & Richardson (2018). SentencePiece. — https://aclanthology.org/D18-2012/ [5] Zhang & Sennrich (2019). Root Mean Square Layer Normalization. — https://arxiv.org/abs/1910.07467 [6] Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. — https://arxiv.org/abs/2104.09864 [7] Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. — https://arxiv.org/abs/2305.13245 [8] Shazeer (2020). GLU Variants Improve Transformer. — https://arxiv.org/abs/2002.05202 [9] Fedus, Zoph & Shazeer (2021). Switch Transformers. — https://arxiv.org/abs/2101.03961 [10] Dao et al. (2022). FlashAttention. — https://arxiv.org/abs/2205.14135 [11] Peng et al. (2023). YaRN. — https://arxiv.org/abs/2309.00071 [12] Kaplan et al. (2020). Scaling Laws for Neural Language Models. — https://arxiv.org/abs/2001.08361 [13] Hoffmann et al. (2022). Training Compute-Optimal Large Language Models. — https://arxiv.org/abs/2203.15556 [14] Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback. — https://arxiv.org/abs/2203.02155 [15] Rafailov et al. (2023). Direct Preference Optimization. — https://arxiv.org/abs/2305.18290 [16] Shao et al. (2024). DeepSeekMath. — https://arxiv.org/abs/2402.03300 [17] Dubey et al. (2024). The Llama 3 Herd of Models. — https://arxiv.org/abs/2407.21783 [18] Meta (2024). Llama 3.1 Community License Agreement. — https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE [19] Jiang et al. (2023). Mistral 7B. — https://arxiv.org/abs/2310.06825 [20] Jiang et al. (2024). Mixtral of Experts. — https://arxiv.org/abs/2401.04088 [21] Mistral AI (2026). Mistral Small 4 Official Model Card. — https://huggingface.co/mistralai/Mistral-Small-4-119B-2603 [22] Qwen Team (2025). Qwen3 Technical Report. — https://arxiv.org/abs/2505.09388 [23] Qwen Team (2025). Qwen3 Official Repository. — https://github.com/QwenLM/Qwen3 [24] Qwen Team (2025). Qwen3-Next Official Model Card. — https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct [25] DeepSeek-AI (2024). DeepSeek-V2. — https://arxiv.org/abs/2405.04434 [26] DeepSeek-AI (2024). DeepSeek-V3 Technical Report. — https://arxiv.org/abs/2412.19437 [27] DeepSeek-AI (2025). DeepSeek-R1. — https://arxiv.org/abs/2501.12948 [28] DeepSeek-AI (2025). DeepSeek-V3.2. — https://arxiv.org/abs/2512.02556 [29] Gemma Team (2024). Gemma 2. — https://arxiv.org/html/2408.00118 [30] Gemma Team (2025). Gemma 3 Technical Report. — https://arxiv.org/html/2503.19786 [31] Google (2026). Gemma 4 Model Card. — https://ai.google.dev/gemma/docs/core/model_card_4 [32] Abdin et al. (2024). Phi-4 Technical Report. — https://arxiv.org/html/2412.08905 [33] OLMo Team (2025). 2 OLMo 2 Furious. — https://arxiv.org/abs/2501.00656 [34] Ai2 (2025). Olmo 3. — https://arxiv.org/abs/2512.13961 [35] Ai2 (2026). Olmo Hybrid: When Linear Attention Meets Transformer. — https://arxiv.org/abs/2604.03444 [36] Z.ai (2025). GLM-4.5. — https://arxiv.org/abs/2508.06471 [37] Z.ai (2026). GLM-5. — https://arxiv.org/abs/2602.15763 [38] Z.ai (2026). GLM-5.2 Official Model Card. — https://huggingface.co/zai-org/GLM-5.2 [39] Moonshot AI (2025). Kimi K2. — https://arxiv.org/abs/2507.20534 [40] Moonshot AI (2025). Kimi K2 Thinking Official Model Card. — https://huggingface.co/moonshotai/Kimi-K2-Thinking [41] Young et al. (2024). Yi: Open Foundation Models by 01.AI. — https://arxiv.org/abs/2403.04652 [42] 01.AI (2024). Yi Apache-2.0 License. — https://github.com/01-ai/Yi/blob/main/LICENSE [43] BigScience (2022). BLOOM. — https://arxiv.org/abs/2211.05100 [44] Biderman et al. (2023). Pythia. — https://arxiv.org/abs/2304.01373 [45] Almazrouei et al. (2023). The Falcon Series of Open Language Models. — https://arxiv.org/abs/2311.16867 [46] MosaicML (2023). MPT-7B Official Model Card. — https://huggingface.co/mosaicml/mpt-7b/blob/main/README.md [47] Databricks (2024). DBRX Official Model Card. — https://huggingface.co/databricks/dbrx-base