Tokenizers from the Ground Up: The Internals and Comparison of BPE, WordPiece, Unigram, and SentencePiece

Start with the problem: the model can't read "words," it only eats integers

Dig to the foundation first. An LLM's first layer is an embedding lookup — as you saw in the earlier PEFT article, it's a V × d matrix, and you use an integer index to pull out the corresponding row vector. The question is: where does the index come from? There is no such thing as a "string" inside the model; the word "resistor" must first become an integer (or a sequence of them) before the table can be looked up. The thing that cuts the human text stream into pieces and maps each to an integer ID is the tokenizer.

This step looks trivial but is the throat of the whole model, because every decision the tokenizer makes ripples downstream:

• Vocabulary size V: directly sets how big the embedding matrix and the output softmax (both V × d) are — larger V means more parameters and memory.
• Granularity of the split: determines how many tokens one sentence becomes. Attention costs O(n²) in sequence length, so doubling token count quadruples the compute. Token count also is your context spend and your API bill.
• Representability: what happens when a word meets something not in the vocabulary? Is the information preserved intact, or does it collapse into an [UNK] black hole?

So "how to split" isn't a preprocessing detail — it's a core design that moves parameter count, compute cost, multilingual ability, and information fidelity all at once. This article dissects everything from the two dumbest splits to the algorithms modern LLMs actually use, down to the level where you could rebuild each yourself.

Two extremes, and why neither works

To understand what modern tokenizers look like, first see the two pits they're avoiding.

Extreme one: word-level (split by "word"). Most intuitive — split on whitespace, one ID per word. But it's sick from the bottom in three ways:

• Vocabulary blowup: English inflections (run / running / ran / runs), compounds, proper nouns, part numbers — each takes a slot, and the vocabulary easily hits hundreds of thousands. A large V bloats the embedding and softmax matrices.
• The OOV (Out-Of-Vocabulary) black hole: a word not seen during training can only map to an [UNK] token at inference. A part number like RC0402FR-0710KL, once absent from the vocab, collapses into a single UNK and the model learns nothing from it.
• Blind to internal structure: running and run are two unrelated IDs in the vocab, so the model must learn each meaning from scratch and can't capture regularities like "-ing is the same suffix." Chinese, Japanese, Thai have no spaces, so word-level stalls outright.

Extreme two: character-level (split by character). Treat each character as a token. It cures the above exactly: tiny vocabulary (a few dozen for English, bounded even with Unicode), never OOV (every word is made of characters). But it buys a deadlier problem:

• Sequence explosion: resistor becomes 8 tokens. Sequence length balloons several-fold, and since attention is O(n²), long sequences wreck compute and memory, while the actual content you can fit in context shrinks drastically.
• Each token carries almost no meaning: a single character r means little, so the model spends huge capacity relearning "which characters together form words," dumping the whole burden of word formation back onto itself.

        Vocab size       Seq length      OOV        Semantic density
       ┌──────────┬───────────────┬──────────┬──────────┐
word   │ blowup ✗  │  shortest ✓    │ severe ✗  │ highest ✓ │
char   │ smallest ✓│  exploded ✗    │ none ✓    │ lowest ✗  │
       └──────────┴───────────────┴──────────┴──────────┘
subword│ moderate ✓│  moderate ✓    │ near-none✓│ moderate ✓│  ← goal: sit in the middle

Read that table and the whole tokenizer field's through-line emerges: we want a split that sits in the middle — frequent words stay whole (high semantic density, short sequences), only rare words break into meaningful pieces (avoiding OOV, controlling vocab size). This is subword tokenization, and every mainstream algorithm below approaches this balance point differently.

The rough cut before the cut: pre-tokenization

Before the subword algorithms, one often-skipped but crucial preliminary step. Most tokenizers don't let the subword algorithm face a whole raw text; they first do pre-tokenization: usually a rule or regex that splits the text into coarse "word" chunks along whitespace and punctuation, and the subword algorithm operates only inside each chunk.

Why this step? Two underlying reasons: one, it bounds the search — subword merges happen only within a word, never gluing two unrelated words across a boundary; two, how whitespace is handled. Whitespace is information ("a" vs " a" differ) and must be preserved. Here approaches fork, planting the seed for SentencePiece later: GPT-2's byte-level encodes whitespace into the token (a leading space becomes the special symbol Ġ), while SentencePiece replaces whitespace with ▁ (U+2581). Remember this fork; we'll pick it up.

Algorithm one: BPE — start from characters, greedily merge the most frequent adjacent pair

The first, and today's most mainstream, subword algorithm is BPE (Byte-Pair Encoding), which Sennrich et al. (2016) borrowed from an old data-compression algorithm and applied to NMT[1]. Its spirit is bottom-up: grow the vocabulary greedily by frequency.

What does the training phase do? Broken into clear steps:

1. Initialize the vocabulary with all base characters in the corpus. Each word is spread into a character sequence (the original paper appends a </w> end-of-word marker).
2. Scan the whole corpus, counting every adjacent symbol pair (weighted by word frequency).
3. Merge the most frequent pair into a new symbol, add it to the vocabulary, and record this merge rule.
4. Repeat 2–3 until the vocabulary reaches your target size (e.g. 30k, 50k).

Walking through a mini corpus (numbers are word frequencies):

Corpus: low×5   lower×2   newest×6   widest×3
Spread into chars (</w> omitted):
   l o w        ×5
   l o w e r    ×2
   n e w e s t  ×6
   w i d e s t  ×3

Count adjacent pairs → most frequent is (e,s)=6+3=9  →  merge rule ①: e + s → es
   n e w es t   ×6
   w i d es t   ×3
Now (es,t)=9   →  merge rule ②: es + t → est
   n e w est    ×6
   w i d est    ×3
Then (w,e)=8   →  merge rule ③: w + e → we ...  and so on

(On a tie, implementations break it by a fixed rule, so the result is deterministic.) Training's product is an ordered list of merge rules — es, est, we… order matters.

Encoding phase (at inference): take a new word, spread it into characters, then apply the learned merge rules in the order they were learned, one by one, merging whatever can merge until no rule applies. Because the rules are ordered and fixed, the same word always splits identically — BPE is deterministic. Frequent words, merged many times during training, end up as single tokens; unseen rare words naturally fall back to a few subwords or even characters, never OOV.

Byte-level BPE — GPT-2's (Radford et al., 2019) key refinement[5]. It swaps the "base characters" from Unicode characters to the 256 bytes. This looks like a technical detail but has big consequences: any string — Chinese, emoji, garbage, any language — is already bytes in a computer, so it can never be OOV, and the [UNK] token need not even exist. The cost: a multi-byte UTF-8 character (a Chinese character is 3 bytes) starts split into several byte tokens and must be regrown via merges, so Chinese has a higher "tokens-per-character" under an English-centric byte-level vocab. GPT-2/3/4, RoBERTa, and Llama 3 all use this route (OpenAI's implementation is tiktoken[9]).

Algorithm two: WordPiece — swap the merge criterion from "most frequent" to "most worthwhile"

WordPiece first came from Schuster & Nakajima (2012) for Japanese/Korean voice search[2], and became widely known when BERT adopted it[6]. Its training framework is nearly identical to BPE (also bottom-up greedy merging); the only, but crucial, difference is the criterion for which pair to merge.

BPE picks the "most frequent" pair. WordPiece picks the pair that most increases the likelihood of the training corpus — which reduces to an easy-to-read score:

              freq(a, b)
score(a,b) = ──────────────────
             freq(a) × freq(b)

Read the formula term by term: the numerator is how often the pair a,b occurs adjacently; the denominator is the product of how often a and b each occur alone. It measures not "how frequent this pair is," but "how much more bound-together a and b are than they'd be if independent." A character that is individually super-frequent (English e) but pairs with everything and sticks to nothing in particular gets its score pushed down by the denominator, so it isn't merged carelessly; conversely, two fragments that are always inseparable yet not individually that frequent score high and get merged first. This biases WordPiece toward growing subwords with "real binding meaning" rather than gluing high-frequency characters everywhere.

WordPiece has a signature design too: mark a word's "continuation piece" with a ## prefix. For example playing splits into play + ##ing; ## tells the model "this piece attaches to the previous one, it's not word-initial." This encodes word-boundary information directly into the token.

The encoding phase also differs from BPE: WordPiece uses left-to-right greedy longest-match-first — from the word start, find the longest prefix in the vocabulary, cut it off, continue from the break, until the word is fully split. The BERT family (BERT, DistilBERT, ELECTRA) all use this.

Algorithm three: Unigram — go the other way, prune down from a big vocab using a probabilistic model

The first two are both "bottom-up merging." Unigram Language Model tokenization, proposed by Kudo (2018)[3], goes the opposite way: top-down pruning, and is a probabilistic model at heart.

Its worldview: assume each subword token has a probability p(token), and the probability of a sentence under a given segmentation is the product of its tokens' probabilities (that's "unigram" — tokens assumed independent). The same word can have many segmentations, each with a probability.

Training uses EM (Expectation-Maximization) to prune iteratively:

1. Build a huge seed vocabulary heuristically (e.g. all frequent substrings, or a very large BPE vocab) — deliberately over-provisioned.
2. Fixing the current vocab, estimate each token's probability and compute the corpus's total probability (loss) under its best segmentation.
3. For every token in the vocab, compute "how much the total corpus probability would drop if it were removed" — its contribution.
4. Delete the batch of lowest-contribution tokens (e.g. the bottom 10%–20%).
5. Repeat 2–4 until the vocab shrinks to the target size.

Two levels of fundamental difference from the first two: opposite direction (prune vs merge), and it is probabilistic — it gives not one segmentation but a probability per segmentation. At encoding time the Viterbi algorithm finds the highest-probability segmentation among all possibilities.

This probabilistic nature unlocks a capability BPE/WordPiece can't natively do: subword regularization. Since a word has multiple reasonable segmentations, training can randomly sample different ones as data augmentation, exposing the model to multiple splits of the same word for robustness[3]. (BPE later added an analog called BPE-dropout.) Unigram is common in T5, ALBERT, XLNet, mBART — especially multilingual models.

SentencePiece — not an algorithm, but a framework that "swallows raw text directly"

Here, dispel a very common confusion: SentencePiece is not a tokenization algorithm but a tool/library, proposed by Kudo & Richardson (2018)[4]. It implements both BPE and Unigram — so "LLaMA uses SentencePiece" is an unfinished sentence; you must ask whether BPE or Unigram runs underneath.

So what is SentencePiece's real innovation? How it handles input. As noted, traditional BPE/WordPiece must first pre-tokenize, and pre-tokenization assumes "words are separated by whitespace" — which fails outright for space-less languages like Chinese, Japanese, Thai. SentencePiece's approach: swallow the entire input as a stream of raw Unicode (including whitespace), treating whitespace itself as an ordinary character escaped to ▁ (U+2581).

This one decision yields two key properties:

• Language-independent: no whitespace-segmentation assumption, treating Chinese and Japanese the same, no need for a different segmenter per language.
• Fully reversible, lossless: because whitespace is preserved as ▁, detokenizing just replaces ▁ back with whitespace to perfectly reconstruct the original text, losing nothing. The traditional pipeline, where whitespace is eaten during pre-tokenization, is harder to reverse losslessly.

Picking up the earlier thread: at the pre-tokenization fork of "how to handle whitespace," SentencePiece chose "tokenize whitespace too (▁)," which is exactly why it can be language-independent and reversible. LLaMA 1/2, T5, and ALBERT all use SentencePiece.

What modern mainstream LLMs actually use

Map the parts above onto real models and you see the industry has converged clearly:

Model family      Underlying algorithm    Framework/impl   Vocab size
────────────────────────────────────────────────────────────────
BERT / ELECTRA    WordPiece               (HF tokenizers)  ~30k
T5 / ALBERT       Unigram                 SentencePiece    ~32k
GPT-2             byte-level BPE          tiktoken         ~50k
GPT-3.5 / GPT-4   byte-level BPE          tiktoken(cl100k) ~100k
LLaMA 1 / 2       BPE                     SentencePiece    32k
Llama 3           byte-level BPE          tiktoken-style   128k

There's a clear trend: vocabularies keep getting bigger. Llama 3 jumped straight from LLaMA 2's 32k to 128k[8]. Why? A bigger vocab → the same text is represented with fewer tokens → shorter sequences (saving O(n²) attention, fitting more content in the same context), and lower token fertility for multilingual text (non-English no longer shredded). The cost is larger embedding and softmax matrices, but for big models that little extra parameter count is a great trade for shorter sequences. Llama 3 also switched from SentencePiece BPE to byte-level BPE (tiktoken-style), for byte-level's "zero OOV + multilingual robustness."

Cross-comparison table

Putting the four splits side by side on the same dimensions (code fence for alignment):

Dimension        BPE             byte-level BPE   WordPiece         Unigram
──────────────────────────────────────────────────────────────────────────────
Base unit        Unicode char    256 bytes        Unicode char      substrings(seed vocab)
Build direction  bottom-up merge bottom-up merge  bottom-up merge   top-down prune
Selection crit.  most freq pair  most freq pair   likelihood score  contribution to corpus prob
Encoding         replay merges   replay merges    left longest-match Viterbi best segmentation
Unique split     yes(determ.)    yes(determ.)     yes(determ.)      no(can sample many)
OOV handling     fall to chars   never OOV(no UNK) fall to chars/[UNK] fall to chars
Boundary marker  (impl-dependent) Ġ leading space ## continuation   ▁ (with SentencePiece)
Data augmentation BPE-dropout    BPE-dropout      none              subword reg (native)
Representative   (early NMT)     GPT/RoBERTa      BERT family       T5/ALBERT/XLNet

One line to catch the difference: BPE/WordPiece are "bottom-up greedy merging," differing only in merge criterion (frequency vs likelihood gain) and boundary marker; Unigram is "top-down probabilistic pruning," gaining sample-able regularization; byte-level swaps BPE's foundation to bytes to eradicate OOV; SentencePiece is the outer framework that can run either BPE/Unigram while being language-independent and reversible.

Downstream consequences of these choices (engineering view)

With the internals covered, on to what you actually hit:

Vocabulary size is a tug-of-war. A big vocab shortens sequences, saves attention, and is fairer multilingually, but the two V × d matrices (embedding and output softmax) grow and eat memory. This is the core trade-off when designing a tokenizer — no free lunch.

Token fertility (tokens per character) varies by language and vocab, reflected directly in cost. Because APIs bill per token, and an English-centric vocab shreds Chinese, code, and rare symbols, the same Chinese passage often costs several times the tokens of equal-length English — for the same sentence, Chinese users pay more and burn context faster. Llama 3 / GPT-4 enlarging their vocabs is largely about squeezing this cost.

How digits and code are split affects capability. Some tokenizers make each digit its own token, others treat 1234 as one chunk — this affects the model's arithmetic, a very real pitfall.

And the "glitch token" phenomenon. The vocab may contain tokens that barely truly appeared in the training corpus (built in by statistics but under-trained); once triggered, the model behaves unpredictably. This reminds us: tokenizer and training corpus must match — the vocab doesn't come from nowhere.

Two concrete reminders for your manufacturing / RAG scenario:

• A part number like RC0402FR-0710KL gets shredded under an English-centric BPE (letters, digits, hyphen each their own chunk), inflating token count while each piece's embedding carries weak semantics — this makes RAG vector retrieval miss part-number semantics. The pragmatic move: actually test how your tokenizer splits typical part numbers, and consider a domain vocabulary or normalizing part numbers if needed.
• For Chinese MES fields, confirm the underlying tokenizer is byte-level or SentencePiece to avoid OOV; also be aware Chinese has high token fertility under a small English vocab, amplifying the cost of long-document retrieval and LLM calls.

Closing: one logic chain tying the whole article together

From start to finish it's one causal chain:

The model only eats integers, so the first step must turn text into token IDs → word-level fails on vocab blowup, the OOV black hole, and blindness to word formation; char-level fails on exploded sequences and thin semantics → so we need a middle-sitting subword that keeps frequent words whole and only breaks rare words → BPE grows an ordered set of merge rules "bottom-up, greedy by frequency," deterministic at encoding, never OOV → byte-level BPE swaps the foundation from characters to 256 bytes, eradicating OOV and dropping even UNK → WordPiece swaps the merge criterion from "most frequent" to a "likelihood-gain score" for pickier merges, marking boundaries with ## → Unigram goes the other way, pruning down with a probabilistic model and Viterbi, unlocking subword regularization → SentencePiece is not an algorithm but a framework that swallows raw text (whitespace as ▁) directly, runs either BPE/Unigram, and is language-independent and reversible → modern LLMs converge to BERT=WordPiece, T5=Unigram, GPT/Llama 3=byte-level BPE, with vocabularies growing to shorten sequences and strengthen multilingual coverage → these choices ripple all the way to parameter count, compute cost, API bills, and multilingual fairness.

One line for the whole article: the tokenizer is the throat between text and model, choosing a balance point among "vocab size, sequence length, semantic density, OOV"; subword is that balance point, and BPE / WordPiece / Unigram / SentencePiece merely approach it with different combinations of "merge vs prune, frequency vs probability, character vs byte."

Extension hooks

To go deeper, pick from: exactly how BPE-dropout and Unigram subword regularization sample during training and how much they help downstream; why tiktoken is 3–6× faster than other implementations (Rust + regex pre-splitting engineering); the scaling relationship between vocab size and final model performance; and, in your RAG pipeline, "how the tokenizer's split affects consistency between embedding retrieval and BM25 segmentation." Any one could be its own article.

References

[1] Sennrich, R., Haddow, B., Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units (BPE). — https://arxiv.org/abs/1508.07909

[2] Schuster, M., Nakajima, K. (2012). Japanese and Korean Voice Search (origin of WordPiece). — https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37842.pdf

[3] Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Unigram LM). — https://arxiv.org/abs/1804.10959

[4] Kudo, T., Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. — https://arxiv.org/abs/1808.06226

[5] Radford, A. et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2 / byte-level BPE). — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

[6] Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (adopts WordPiece). — https://arxiv.org/abs/1810.04805

[7] Touvron, H. et al. (2023). LLaMA: Open and Efficient Foundation Language Models (SentencePiece BPE, 32k). — https://arxiv.org/abs/2302.13971

[8] Grattafiori, A. et al. (2024). The Llama 3 Herd of Models (switches to 128k byte-level BPE). — https://arxiv.org/abs/2407.21783

[9] OpenAI (2023). tiktoken: a fast BPE tokeniser for OpenAI models. — https://github.com/openai/tiktoken