▚ TSAI_CHENG-HUNG▌
← ALL POSTS
LOG_ENTRY · Jul 13, 2026 · ⊙ 23 MIN READ

Word Segmenters from the Ground Up: The Paradigms Behind jieba, the Mainstream Tools, and How They Compare (vs. Subword Tokenizers)

jieba is a segmenter, but on a different level from a subword tokenizer. Chinese has no spaces and splits are ambiguous, so segmentation is fundamentally disambiguation. This article converges segmentation into four underlying paradigms — maximum matching, dictionary+statistics (jieba's DAG+DP+HMM), sequence labeling (HMM→CRF→BiLSTM-CRF→BERT), and Japanese lattice (MeCab) — dissects each mechanism, surveys and compares mainstream tools (jieba/THULAC/pkuseg/HanLP/LTP/CkipTagger/MeCab), and ties back to the previous article on how segmenters and subword tokenizers coexist complementarily in RAG.

#Tokenizer#Word Segmentation#jieba#NLP#Deep Dive

Start with the problem: Chinese has no spaces, so word boundaries must be found

Dig the problem to the bottom first. English writing natively separates words with whitespace — the resistor is hot is obviously four words. But Chinese, Japanese, and Thai are so-called scriptio continua: 中央處理器過熱 ("the CPU overheats") has no delimiter anywhere. Yet to do keyword search, build a BM25 inverted index, tag parts of speech (POS), or extract named entities (NER), the first step is always knowing "where are the word boundaries in this character stream." Cutting a continuous text into individual linguistic "words" is called word segmentation, and jieba is a tool that does exactly this.

And it's harder than it looks, because the split is ambiguous. A classic example:

「研究生命科學」("study life science")
   split A：研究 / 生命 / 科學    ← study / life / science  ✓
   split B：研究生 / 命 / 科學    ← grad-student / life / science  (nonsense)
「乒乓球拍賣完了」
   split A：乒乓球 / 拍賣 / 完了   ← ping-pong balls / auctioned / off
   split B：乒乓 / 球拍 / 賣完 / 了 ← ping-pong / paddles / sold-out

Both splits are "legal" on the surface (every chunk is a dictionary word), but only one is reasonable in context. So word segmentation isn't as simple as "look up and cut" — it's fundamentally a disambiguation problem: among all legal segmentations, pick the most reasonable one. That sentence is the whole article's foundation — you'll see that every segmenter's evolution is about doing this disambiguation more accurately.

Let's fix the positioning first: this article is about word segmentation, which is a different level from the subword tokenization covered in the previous article Tokenizers from the Ground Up. They're often conflated; I'll devote a section at the end to making the difference crisp. For now, remember: word segmentation cuts out "real words," while a subword tokenizer cuts out "tokens in a model's vocabulary."

Set up the frame first: the four underlying paradigms of segmentation

There are many tools on the market, but the underlying mechanisms converge into four paradigms. Set up the frame first, tag each tool by which one it belongs to, and you won't drown in tool names:

Paradigm①  dictionary + rules      maximum matching (greedy longest word)  — oldest, fastest, most brittle
Paradigm②  dictionary + statistics  build DAG + max-prob path by word freq  — jieba precise mode
Paradigm③  sequence labeling        recast as "tag each char B/M/E/S"       — HMM→CRF→BiLSTM-CRF→BERT
Paradigm④  morphological + lattice  dictionary lattice + lowest-cost path   — Japanese MeCab

Paradigms ①② rely on a dictionary; paradigm ③ recasts segmentation as a machine-learning classification problem (the modern backbone); paradigm ④ is Japanese's fused version. Let's dissect each to the bottom.

Paradigm ①: dictionary + maximum matching — most intuitive, most brittle

The oldest approach is Maximum Matching, relying purely on a dictionary + one greedy rule. Forward Maximum Matching (FMM) works like this: start from the leftmost of the sentence, find "the longest word in the dictionary matchable from this position," cut it off, continue from the break, until done. Backward Maximum Matching (BMM) does the same right-to-left. Bi-directional maximum matching runs both and picks one by heuristics (which yields fewer words / fewer single chars).

Dict = {中央, 中央處理器, 處理, 處理器, 過熱}
Sentence = 中央處理器過熱

FMM: find longest from left → "中央處理器"(5 chars, in dict) → cut!
     remaining "過熱" → "過熱" → cut!
     result: 中央處理器 / 過熱   ✓

The underlying logic is just "look up + greedily take the longest," with no probability, no context judgment. Advantages: extremely fast, extremely simple, fully controllable (the dictionary rules). But it's mechanically flawed in three ways:

So maximum matching today is only a fallback or ultra-lightweight option. To disambiguate, you must introduce probability — which is what jieba does.

Paradigm ②: dictionary + statistics — jieba's core

jieba is today's most popular Chinese segmenter[1], and its precise mode is the representative of paradigm ②, realizing "disambiguation" with probability. It works in three steps — the first two handle in-dictionary, the third fishes out-of-dictionary:

Step 1: build a DAG with a prefix dictionary. When jieba loads the dictionary (each entry is word + frequency), it builds a prefix dictionary (a Trie) for quickly querying "all dictionary words starting at a position." For a sentence, it scans all possible words into a DAG (directed acyclic graph) — nodes are character positions, each edge means "this span is a dictionary word." For an ambiguous sentence, the DAG contains multiple paths (multiple splits).

Step 2: find the max-probability path with dynamic programming. This is the key to disambiguation. jieba turns each word's frequency into a (log) probability, then uses dynamic programming (DP) to find, among all split paths, the one with "the largest product of word probabilities across the whole sentence."

DAG (partial) of「研究生命科學」:
   研究 ─── 生命 ─── 科學      path A: P(研究)·P(生命)·P(科學)
   研究生 ── 命 ──── 科學      path B: P(研究生)·P(命)·P(科學)
   DP compares the probability products → picks the larger → usually A wins ✓

See why it beats maximum matching: maximum matching blindly takes the local longest, while jieba makes a global choice across all splits using the whole-sentence joint probability. That's disambiguation — it asks not "which word is longest" but "which segmentation is most probable overall."

Step 3: fish out unknown words with an HMM. What about out-of-dictionary character runs (new words, names, part numbers)? jieba fires up an HMM (Hidden Markov Model): tag each character with one of four hidden states B/M/E/S (Begin / Middle / End / Single-char word); the model has learned, on annotated corpora, "transition probabilities" (e.g. B is likely followed by M or E) and "emission probabilities" (probability of a character given a state), then uses the Viterbi algorithm to solve for the most probable state sequence, assembling B→E or B→M→E into new words.

Here's the tie-back to the previous article: jieba's DP and the HMM's Viterbi are mathematically the same lineage as the Viterbi in that article's Unigram — both "find, among exponentially many splits, the one with highest joint probability." The difference is where the probabilities come from: jieba's word probabilities come from a human dictionary's frequencies, Unigram's come from probabilities learned via EM on a corpus. The same mathematical tool, fed different probability sources, one serving segmentation, one serving subword tokenization.

Paradigm ③: sequence labeling — recast segmentation as "tag each character"

This is the backbone of modern segmentation, and the article's most important level insight. Xue (2003) pointed out something: segmentation can be equivalently recast as a "per-character classification" problem[2]. How? Give every character in the sentence a label indicating its position within a word:

char:   中  央  處  理  器  過  熱
label:  B   E   B   M   E   B   E
        └中央┘ └處理器─┘ └過熱┘
rule:   B(begin) M(middle) E(end) S(single-char word)
        once labels are assigned, boundaries emerge: a cut after every E or S.

This transform is crucial, because it turns "segmentation," a seemingly special problem, into the most thoroughly studied sequence labeling problem in machine learning. So the decades of accuracy gains in segmentation are essentially the history of "the same B/M/E/S labeling task, with ever-stronger models underneath." Let's peel this evolution apart layer by layer:

Generation 1: HMM (generative model). It models "how labels generate characters" — learning emission probabilities P(char|label) and label transitions, solving the most probable label sequence with Viterbi. CAS's ICTCLAS / NLPIR uses a hierarchical HMM (HHMM)[3]. HMM's flaw is its overly strong independence assumption: it assumes each character is generated by the current label alone, making it hard to jointly use rich features like "what's the previous char, the next char, is this a digit, character n-grams."

Generation 2: CRF (Conditional Random Field, discriminative model). CRF is a key leap in segmentation history — Stanford's segmenter[4], Japanese MeCab[5], and Peking University's pkuseg[7] are all CRF. Its three mechanistic advantages over HMM deserve a term-by-term walk:

(Tsinghua's THULAC takes a similar route with a structured perceptron, another discriminative model that also consumes rich features.)

Generation 3: BiLSTM-CRF (neural sequence labeling). CRF is strong, but features must be hand-designed. Huang et al.'s (2015) BiLSTM-CRF[6] automated feature engineering too: turn each character into a character vector, sweep the whole sentence from both sides with a bidirectional LSTM to automatically learn each character's "context-fused" representation (replacing hand features), then stack a CRF layer on top.

Why put a CRF layer atop a neural net? This is a key point many miss: if you only softmax-classify each character independently, the model may emit illegal label sequences (e.g. B directly followed by B, or E followed by M), because per-character independent decisions don't know labels have dependencies. The CRF layer specifically models label-to-label transitions, forcing legal B/M/E/S sequences. So the BiLSTM "understands context" and the CRF "guarantees legal labels" — a clean division of labor.

Generation 4: BERT / Transformer. Swap the BiLSTM for pretrained Transformer character representations (BERT-like), still topped with CRF or softmax for fine-tuning — this is the current SOTA. HIT's N-LTP[8], Han He's HanLP[9], and Academia Sinica's CkipTagger / CKIP Transformers[10] all take this route, pushing contextual understanding to new heights via large-scale pretraining.

One line to close paradigm ③: once segmentation is recast as per-character B/M/E/S labeling, the task frame is fixed; HMM → CRF → BiLSTM-CRF → BERT is a ladder of model capability under the same task, and each rung up brings richer features and deeper context understanding, hence higher accuracy.

Paradigm ④: morphological analysis + lattice — Japanese's MeCab

Japanese also has no spaces, but with an extra complication: it has extensive inflection (verbs and adjectives conjugate), so Japanese does not just segmentation but morphological analysis — cutting morphemes while tagging POS and base-form readings. The representative tool MeCab (Kudo et al., 2004)[5] is a fused upgrade of paradigms ②③:

Spread the sentence into a lattice with the dictionary (fuller than a DAG):
   each candidate morpheme is an edge with a "word cost" (the morpheme's own cost)
   adjacent morphemes have a "connection cost" (POS-adjacency cost, e.g. particle→verb)
        ↓
   all these costs are trained from annotated corpora with CRF
        ↓
   use Viterbi to find the "lowest total cost" path = best morpheme segmentation

You can see it sews two paradigms together: build a lattice with a dictionary (like jieba's DAG) + learn edge and connection costs with CRF (like paradigm ③'s discriminative learning), then Viterbi for the lowest-cost path. The later Juman++ (RNN language model + beam search) and Sudachi (business-grade, adjustable granularity) extend this route.

A survey of mainstream tools (grouped by paradigm)

Spread the varieties out, classified by paradigm and language, and you can tell at a glance who's who:

── Chinese (mainly Simplified) ────────────────────────────────────
jieba          P②   dict+DP+HMM         most popular, light, fast, easy; default model is older
ICTCLAS/NLPIR  P③   hierarchical HMM    veteran (CAS), seg+POS+NER combined
THULAC         P③   structured perceptron  Tsinghua, discriminative, fast & accurate
pkuseg         P③   CRF, multi-domain   PKU, separate web/medicine/tourism models, strong domain fit[7]
LTP / N-LTP    P③   neural + multi-task HIT, one-stop (seg→parse→semantics)[8]
HanLP          P③   neural, dual-engine Han He, production-grade, 130 languages[9]
LAC            P③   neural (Baidu)      seg+POS combined
── Traditional Chinese ────────────────────────────────────────────
CkipTagger     P③   BiLSTM/deep learning  Academia Sinica CKIP, Trad-Chinese, WS+POS+NER[10]
CKIP Transformers P③ BERT-based         CKIP's Transformer version, Trad-Chinese SOTA
── Japanese ───────────────────────────────────────────────────────
MeCab          P④   lattice+CRF         classic morphological analyzer[5]
Juman++        P④   lattice+RNN-LM      factors in language-model score
Sudachi        P④   lattice+granularity business-grade, multi-granularity splits
── Korean ─────────────────────────────────────────────────────────
Mecab-ko/Okt/Komoran  P③④  mostly dict+stats/neural
── English / multilingual ─────────────────────────────────────────
spaCy / NLTK / Moses  P①  whitespace+rules  English has spaces; focus on punctuation, contractions, clitics (don't→do n't)
ICU BreakIterator     P①  Unicode rules  cross-language char/word boundary detection

A practical reminder for Traditional-Chinese users: most Simplified tools (jieba, pkuseg) default to Simplified dictionaries and training corpora, so using them directly on Traditional Chinese loses accuracy; for Traditional Chinese, prefer CkipTagger / CKIP Transformers (Academia Sinica), or swap a Traditional dictionary into jieba.

Cross-comparison table

Putting representative tools on the same dimensions (code fence for alignment):

Tool        Language  Paradigm  Core method       OOV ability  Speed  Domain fit  Typical use
──────────────────────────────────────────────────────────────────────────────────────────
Max Match   general   ①         dict+greedy       none         v.fast swap dict   ultra-light/fallback
jieba       Ch(Simp)  ②         DAG+DP+HMM         medium(HMM)  fast   weak(custom)  general, fast prototype
THULAC      Ch(Simp)  ③         struct. perceptron medium       fast   medium        balance accuracy/speed
pkuseg      Ch(Simp)  ③         CRF+multi-domain   med-high     medium strong(by dom) medical/legal verticals
HanLP/N-LTP multi     ③         BERT/neural mtask  high         slow   high(tunable) production, high accuracy
CkipTagger  Ch(Trad)  ③         BiLSTM/BERT        high         medium medium        Trad-Chinese WS/POS/NER
MeCab       Japanese  ④         lattice+CRF        medium       v.fast medium        Japanese morph. standard
spaCy       En/multi  ①         rules+statistics   —(has spaces) fast  high          English/multi NLP pipeline

Key takeaways: want fast and light with acceptable accuracy → jieba; want domain accuracy → pkuseg (by domain) or fine-tune HanLP; want Traditional Chinese → CkipTagger; want Japanese → MeCab; want multilingual production-grade → HanLP. The further toward paradigms ③④, the higher the accuracy but the slower and more resource-hungry — the universal trade-off in segmentation.

Tie-back to the previous article: segmentation vs subword tokenizer, what's the difference

Now connect the two articles. Many confuse jieba and BPE because both are called "分詞" in Chinese, but they're different levels with different purposes:

Dimension        word segmentation (here, e.g. jieba)   subword tokenization (prev, e.g. BPE)
────────────────────────────────────────────────────────────────────────────────────
What's cut out    linguistic "words" (human-readable)   model-vocab tokens (maybe not words, e.g. ##ing)
Product           word strings                          integer IDs (for embedding lookup)
Vocab source      human dictionary + annotated training unsupervised corpus stats (merge/prune)
Core goal         disambiguation: pick the best split   efficiency: balance vocab size & seq length
Disambiguates?    yes (the essence of segmentation)     no (usually deterministic, no linguistic aim)
OOV               guessed via HMM/neural, still errs     byte-level never OOV
Bound to a model? standalone tool, model-agnostic       bound to one model's vocabulary
Serves            traditional retrieval/BM25/POS/NER/humans  neural nets (LLM/BERT/GPT)

The key is they're complementary, not conflicting, and often coexist in your RAG pipeline:

This neatly closes the hook planted at the end of the previous article — "how the tokenizer's split affects consistency between embedding retrieval and BM25 segmentation." The answer: these two branches use two completely different splitting systems; when doing RRF fusion in hybrid search, be aware that the BM25 branch's granularity (words) and the vector branch's granularity (subword tokens) are inherently different, and part numbers and proper nouns get split differently on each side — a common pitfall when tuning hybrid retrieval.

Closing: one logic chain tying the whole article together

Chinese has no spaces, and the split is ambiguous → so segmentation is fundamentally disambiguation (pick the most reasonable among legal splits) → maximum matching (dictionary + greedy longest) is simplest but doesn't disambiguate, can't handle OOV, and is blind to context → jieba does global disambiguation with a DAG + word-frequency DP, then fishes unknown words with HMM + Viterbi → more fundamentally, Xue recast segmentation as a "per-character B/M/E/S" sequence-labeling problem → the task frame is now fixed, and the rest is swapping stronger models: HMM (generative, weak features) → CRF (discriminative, global normalization, arbitrary features) → BiLSTM-CRF (auto-learned context features, CRF guarantees legal labels) → BERT (pretrained representations, SOTA) → Japanese MeCab sews dictionary and discriminative learning together with lattice + CRF → the various tools (jieba light, pkuseg domain, CkipTagger Traditional Chinese, HanLP multilingual production) are just different trade-offs of these paradigms → finally the tie-back: segmentation yields "words" serving traditional retrieval and humans; subword tokenizers yield "token IDs" serving neural nets, and the two coexist complementarily in RAG hybrid retrieval.

One line for the whole article: segmentation is "cutting continuous text into linguistic words, fundamentally a disambiguation problem," and its tech history is the history of the same B/M/E/S labeling task with the model swapped from HMM all the way to BERT; it is not the same thing as the previous article's subword tokenizer, but a coexisting second splitting system in your RAG pipeline.

Extension hooks

To go deeper, pick from: what CRF feature templates actually look like and how much they affect accuracy; why in the BERT era many skip explicit segmentation and let downstream tasks eat characters (char-based) or subwords directly; the pitfalls of ordering between Traditional/Simplified conversion and segmentation; and, in your hybrid retrieval, whether to align the BM25 branch's segmentation granularity to the vector branch's tokenizer (or vice versa). Any one could be its own article.

References

[1] fxsjy (2012–). jieba Chinese text segmentation (prefix dict + DAG + DP + HMM/Viterbi). — https://github.com/fxsjy/jieba

[2] Xue, N. (2003). Chinese Word Segmentation as Character Tagging. — https://aclanthology.org/O03-4002/

[3] Zhang, H.-P. et al. (2003). HHMM-based Chinese Lexical Analyzer ICTCLAS. — https://aclanthology.org/W03-1730/

[4] Tseng, H. et al. (2005). A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005 (Stanford, CRF). — https://aclanthology.org/I05-3027/

[5] Kudo, T., Yamamoto, K., Matsumoto, Y. (2004). Applying Conditional Random Fields to Japanese Morphological Analysis (MeCab). — https://aclanthology.org/W04-3230/

[6] Huang, Z., Xu, W., Yu, K. (2015). Bidirectional LSTM-CRF Models for Sequence Tagging. — https://arxiv.org/abs/1508.01991

[7] Luo, R. et al. (2019). PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation. — https://arxiv.org/abs/1906.11455

[8] Che, W. et al. (2021). N-LTP: An Open-source Neural Language Technology Platform for Chinese. — https://arxiv.org/abs/2009.11616

[9] He, H. (2020–). HanLP: multilingual NLP toolkit for production. — https://github.com/hankcs/HanLP

[10] CKIP Lab, Academia Sinica (2019–). CkipTagger: Traditional Chinese word segmentation / POS / NER. — https://github.com/ckiplab/ckiptagger

Word Segmenters from the Ground Up: The Paradigms Behind jieba, the Mainstream Tools, and How They Compare (vs. Subword Tokenizers) — Tsai Cheng-Hung