Which Pieces Does Unigram Tokenization Really Need?
arXiv:2512.12641v2 Announce Type: replace
Abstract: The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practice is complex, limi…